JP5913145B2 - Log visualization device, method, and program - Google Patents

Description

  The present invention relates to a log visualization apparatus, method, and program, and more particularly, to a log visualization apparatus, method, and program for visualizing information useful to a user from log information output by various devices. Specifically, the present invention relates to a technique suitable for visualizing necessary information without a user knowing in advance a generation rule of a log format generated by a device.

  Today, centrally monitoring and managing devices and software with different roles from different manufacturers, mainly due to cost savings. On the other hand, each of these various devices and software has a mechanism for outputting a log having a unique format, and is used when monitoring and managing the devices. With the development of information equipment, the log information has become complicated and large-scale, and an efficient monitoring method is required.

  Under such circumstances, there is an analysis base for simplifying log analysis. The user can monitor the current network status by visualizing the occurrence status of the log message on the base (for example, see Non-Patent Document 1).

  In addition, it targets syslog generated by network devices such as routers, and provides a method for grasping templates when a certain format such as vendor, message type, error code, and detailed message content is given in advance. A method of displaying the digest information of syslog using the positional relationship of routers has been proposed (for example, see Non-Patent Document 2).

Splunk http://www.splunk.com/ T. Qiu, Z. Ge, D. Pei, J. Wang, J. Wang, J, Xu, "What Happened in my Network? Mining Network Events from Router Syslog", In Proc. Of IMC, 2010.

  However, the visualization that is possible with the technique of Non-Patent Document 1 described above relates to the parameters in the log, the statistic of the target character string, and the time transition by specifying the log format and the character string included therein. However, it is not suitable for temporary visualization of large-scale and complex logs generated from devices of different roles and manufacturers.

  Further, the technology of Non-Patent Document 2 requires at least prior knowledge at the stage of collecting information. For example, it is necessary to know from which vendor the log is collected and which part of the message indicates the error code. Manual input and preparation are required. In addition, since the digest display function of syslog is a technique specialized for syslog, it cannot be applied to other monitoring log information. Moreover, although the obtained information is digest information, it is a text message, which is insufficient to obtain an intuitive interpretation when performing network monitoring.

  The present invention has been made in view of the above points, and a user can effectively generate an enormous log message generated by a machine in which a plurality of formats coexist without depending on a specific format. It is an object of the present invention to provide a log visualization apparatus, method, and program capable of visualizing the entire log.

In order to solve the above problem, the present invention (Claim 1) is a log visualization device for visualizing log information of a plurality of monitoring target devices,
Log information storage means for storing log information including device identification information for each device to be monitored and a monitoring device label given by an additional attribute of the device to be monitored;
A log that stores template information in which an important part extracted from the log information (hereinafter referred to as “log template”) is given a label (hereinafter referred to as “template label”) according to the usage status. Template storage means;
Log group storage means for storing a log group in which the template information is grouped based on the co-occurrence of log information corresponding to the template information ;
And said log information before Symbol log information storage unit, performs matching between the log template storage means or information of the log group storage unit, a log group, a log template label ordinate, the time of occurrence of the log information to the horizontal axis Graph generating means for generating a graph by plotting and outputting the graph to a display device.

Further, according to the present invention (Claim 2), in the graph generating means, the vertical axis of the graph is
Mapping the log template and the log group so as to take a higher value as the frequency of the log group is lower.
Mapping so as to take a higher value as the appearance order of the log template and the log group is slower,
Mapping the log template and the log group with a low periodicity value so as to take a high value,
Mapping so that the log template and log group that are similar are close
Define and map the order relationship between the log template and the log group in advance.
Means for drawing the graph in combination of at least one or a plurality of the above.

Further, the present invention (Claim 3) provides the graph generating means,
And means for dividing and displaying the graph for each device type of the monitored device.

The present invention (Claim 4) provides the graph generating means,
Means for changing the display of symbols used for displaying the graph in accordance with the number of log templates generated in the log group or the number of log templates and log groups frequently generated in the same time.

Further, the present invention (Claim 5) provides the graph generating means,
Means for changing the shape, color, and size of the symbol for each device identification information;

  As described above, according to the present invention, an enormous log message generated by a device in which a plurality of formats are mixed by analyzing past log information on a sufficient edge generated by a device group to be monitored by a user. Thus, the entire log can be effectively visualized for the user without depending on a specific format.

1 is a system configuration diagram according to an embodiment of the present invention. It is a log plot utilization flow in one embodiment of the present invention. It is an example of the graph in one embodiment of the present invention. It is a system configuration example (offline use example) of the first embodiment of the present invention. It is a system configuration example (online use example) of the second embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The present invention solves the problem by analyzing a sufficient amount of past information generated by a device group to be monitored by a user.

  FIG. 1 shows a system configuration according to an embodiment of the present invention.

  The log visualization apparatus 10 shown in FIG. 1 includes a user interface 11, a file control unit 12, a graph generation unit 13, a log group DB 14, a log template DB 15, and a log DB 16. An input device 20 and a display device 30 used by the user are connected to the user interface 11.

As shown in Table 1, the log template DB 15 stores a log template for each template ID.

ロググループDB１４には、表２に示すように、ロググループID毎にグループに属するログテンプレートIDが格納されている。なお、ロググループの作成は過去のデータから予めオフラインで行われているものとする。

The log group DB1 4, as shown in Table 2, log template ID that belongs to the group is stored for each log group ID. Note that the log group is created offline in advance from past data.

ログDB１６には、表３に示すように、ログ収集装置４０で収集されたログメッセージとして、ログ生起時間、ログ、ホストID，テンプレートIDが格納されている。

As shown in Table 3 , the log DB 16 stores a log occurrence time, a log, a host ID, and a template ID as log messages collected by the log collection device 40.

ファイル制御部１２は、ログメッセージ内の重要とされる部分（以下、「ログテンプレート」と記す）を抽出し、ユーザインタフェース１１を介して利用者から入力されたラベル（以下、「テンプレートラベル」と記す）を付与し、ログテンプレートDB１５に格納する。ログテンプレートの抽出方法として、対象がsyslogのみではない一般的なログのテンプレートを抽出する方法である。この方法は、ログDB１６からログメッセージを取得して、ログメッセージに含まれる各単語に対して、出現位置と単語・ログメッセージ内の単語数の出現頻度の組に基づいてスコアを算出し、当該スコアに基づいてクラスタリングを行うことによりテンプレートを作成する。このとき、クラスタリングされたクラスタの値の高いものから併合し、ログメッセージの総単語数と重要な単語を含むクラスタの範囲を決定するための所定の閾値を乗じた値を超えた時点において併合した単語の集合をテンプレートとする方法がある。

The file control unit 12 extracts an important part (hereinafter referred to as “log template”) in the log message, and a label (hereinafter referred to as “template label”) input from the user via the user interface 11. Are stored in the log template DB 15. As a log template extraction method, a general log template whose target is not only syslog is extracted. This method obtains a log message from the log DB 16, calculates a score for each word included in the log message based on the combination of the appearance position and the appearance frequency of the number of words in the word / log message, A template is created by performing clustering based on the score. At this time, the clustered clusters are merged starting from the highest value, and merged when the total number of words in the log message exceeds the value multiplied by a predetermined threshold for determining the range of clusters containing important words. There is a method using a set of words as a template.

  The template label indicates a label such as a service type, log priority / importance, etc., depending on the usage status. Note that the template label is given by the user as necessary, and may not be provided.

Further, the file control unit 12 groups each log template according to its co-occurrence using an existing log occurrence rule extraction method or the like, and sets a log template group (hereinafter referred to as “log group”) to the log group DB1. 4 is stored. At this time, the number of log template elements included in the group may be one, and it is allowed to belong to a plurality of groups. As a log occurrence rule extraction method, there is a method of automatically grasping a log occurrence rule by paying attention to the co-occurrence of logs. This method extracts a template corresponding to a log that occurred in the past, clusters the templates with high co-occurrence for each monitored device, generates a group of monitored devices and templates, and uses the group as an event. There is a method in which the number of occurrences is counted with respect to the monitored device and template of the event, and an occurrence rule having a high number of occurrences is used.

  It is assumed that the log group grasped in advance grasps the time when the log group has appeared so far, and also stores the time in the log group DB 14.

  The log collection device 40 generates a log and stores it in the log DB 16. It is assumed that the log includes information (for example, device ID) indicating the monitoring target device 50 that generated the log, and the monitoring target device 50 can be specified. Further, it is assumed that the information on the devices to be monitored 50 is labeled with additional attributes such as a service type, a position, and a device type according to the usage situation. These labels are given by the user as needed, and may be omitted. Hereinafter, this is referred to as a “monitoring device label”.

  The graph generation unit 13 first checks which log template and log group the log information generated in the past stored in the log DB 16 matches by matching the log template DB 15 and the log group DB 14, and displays this graph. Is drawn on the display device 30 via the user interface 11.

  In the following, the method of the present invention will be described in detail using the procedure diagram of FIG.

  It is assumed that a unique monitoring device ID is assigned to each monitoring target device 50 in advance, and similarly, a log template ID is assigned to each log template.

  Step 101) First, the user designates a log time series to be visualized from the input device 20. Specifically, a time and a time width are input.

  Step 102) Subsequently, the service providing type, the monitoring device group type, the area name, the corresponding customer type, and the monitoring device label to be visualized are received from the input device 20. The monitoring device label indicates information related to the monitoring device such as a vendor, an area, a target service, and a protocol.

  Subsequently, the user can select any rule from the input device 20 such as various conditions such as a log template label that matches an arbitrary rule (for example, including a specific word), a log template occurrence frequency, and a log template periodicity. Specify a log template that matches the above, and specify whether these are to be displayed or not. The log template label represents the urgency level, service, target part, etc. of the log template.

  Step 103) The user designates a filter based on the frequency of occurrence of the log template (such as the number of seconds generated in the total observation time). Since a log that occurs frequently in a long time series is considered to have low anomaly, filtering this makes it easier to visually recognize the abnormal series. This occurrence frequency can be acquired as a value within an arbitrary time width as well as within the observation time.

  Step 104) Specify a filter based on the periodicity of the log template. As an example, log messages generated by cron jobs that occur periodically every day are low in anomaly, but cannot be removed by a frequency filter because the frequency is low. Therefore, filtering focusing on periodicity is performed. There are many methods for calculating the periodicity, but as an example, there is a method using a coefficient of variation of the number of occurrences per day as follows. This periodicity can be acquired not only within the observation time but also as a value within an arbitrary time width.

  For example, the number of occurrences of a unique {log template, generated monitoring device} combination is calculated at certain time intervals such as one day, and several samples such as one week are collected. By calculating the coefficient of variation of the element of the obtained number of occurrences, it is possible to calculate how much the number of occurrences varies at each time interval.

  Step 105) Subsequently, the graph generation unit 13 performs matching between the log of the log DB 16 and the log template of the log template DB 15, or the log and the log group of the log group DB 14, and which log template or which log group at which time. To know what happened.

  As shown in FIG. 3, the graph generation unit 13 uses the log group and log template ID as the vertical axis and the time when the log occurred as the horizontal axis. If the corresponding log template or log group occurs, the ID and time Draw a point at a point. At this time, drawing is performed so that a monitoring device that is uniquely generated can be identified using the shape of a color or a point. In addition, about the mapping method to the vertical axis | shaft of a log group and log template ID, the visualization according to a use use is attained by combining the some parameter | index mentioned below.

<Indicator>
(a) Frequency of occurrence: The lower the frequency of log templates and log groups, the higher the value on the vertical axis. Also, this frequency can be set within an arbitrary time width such as within the entire observation time or within the selected time. Further, it is possible to select whether to adopt a value of either the frequency before the filtering or the frequency after the filtering.

  (b) Occurrence order: The values on the vertical axis are acquired in order from the lowest in the order of appearance of log templates and log groups. In this case, the log that occurred later will have a higher vertical axis value, and when pursuing the cause retroactively from a certain abnormal event, the log that occurred immediately before is drawn at a higher position, so it is easier to check There is an advantage.

  (c) Periodicity: Similar to (a), the lower the periodicity value of the log template and log group, the higher the value on the vertical axis. Further, it is assumed that the periodicity can be acquired as a value within an arbitrary time width such as within all observation times and within a selected time. Further, it is possible to select whether to adopt either the periodicity value before filtering or the periodicity value after filtering.

  (d) Long template and log group similarity: A vertical axis that is similar to a similar log template and log group by comparing the words in the log template and log group in advance and grasping the similarity relationship between them. To take. In this case, since an order relationship cannot be formed, a method of using another index as the first sort and using it for the second sort can be considered. There are various methods for grasping the similar relationship between the log template A and the log template B. As an example, there is a method of calculating the cos similarity of words included in A and B. The cos similarity is the number of possible words V included in the log template. If the jth (j = 1, ..., V) word is included in the log template, the jth By assigning to each log template a V-dimensional vector that becomes 1 and others become 0, calculation can be performed for any two vectors. The calculation method of the similarity is not limited to the cos similarity, and various distances such as an edit distance and a jaccard distance can be considered.

  (e) Order given in advance by the user: The log template and the log group can be mapped on the vertical axis by defining the order relation for the log template and the log group in advance by the user.

  Different graphs can be output by combining the indicators (a) to (e). For example, if (a) is used for the first sort and (c) is used for the second sort, the lower the frequency, the higher the position, and the higher the frequency, the higher the frequency. This is effective when tracing the log from a certain abnormal event and pursuing the cause.

  On the other hand, by combining (d) and (a), low frequency occurrences are basically at high positions, but logs with high occurrence frequencies with similar contents can be rendered at low or high positions. it can. This makes it possible to visualize not only the occurrence frequency but also the contents of the log.

  In addition, there are the following methods (f) and (g) as unexpected visualization variations that change the display method of the vertical axis.

  (f) Divide and display the graph for each device type. For example, the monitoring device IDs 1 to 10 are displayed on the screen 1, and the monitoring device IDs 11 to 20 are displayed on the screen 2. The display method in each divided and displayed graph is in accordance with the methods (a) to (e).

  (g) The display size of the symbol used for displaying the graph is changed according to the number of log templates generated in the log group and the number of log templates and log groups frequently generated in the same time. At this time, according to a function (for example, log (x)) that gradually increases when a threshold is set or the number of occurrences is set to x so that symbols are not too large according to a large number of log templates and log groups generated, There are methods such as determining the size.

  (h) By specifying the shape, color, and size of the symbol corresponding to the monitoring device ID specified by the user in advance and displaying it according to this, it is possible to highlight information about the specified monitoring device. is there.

  Step 106) As described above, the graph generation unit 13 displays the drawn graph on the display device 30.

[First embodiment]
FIG. 4 shows the system configuration of the first embodiment of the present invention.

  In the system shown in the figure, a user interface 210 is connected to the log visualization device 200, and a graph generation engine 220 is connected to the user interface 210. The user interface 210 inputs a filter setting value, a configuration method of the vertical axis of the graph, and the like input from the user 100 in advance to the graph generation engine 220 and outputs information acquired from the graph generation engine 220.

  It is assumed that the user 100 is executing the processing from step 101 to step 104 shown in FIG. 2 on the user interface 210 of the log visualization device 200.

  The graph generation engine 220 stores a log group DB 240 that stores a group of templates that are likely to occur at the same time, a log template DB 250 that stores templates of logs that have occurred in the past, and logs collected by the log collection device 230. The log DB 260 is referred to, and the contents are output to the user interface 210.

  In the system configuration example of FIG. 4, it is assumed that logs collected by the log collection device 230 are stored in the log DB 260 in advance. The graph generation engine 220 can output a graph indicating the occurrence of a log of an arbitrary past time as a digest of a log that has occurred in the past by performing the process of step 105 in FIG.

  Specifically, the input log may be from any device, but for example, a router syslog for network monitoring can be considered. In this case, the offline form of FIG. 4 will be described.

  Logs generated by the network devices are collected centrally by the log collection device 230, and templates and monitoring device information prepared in advance are given and stored in the log DB 260.

  The user designates a time series to be analyzed, a group of monitoring devices, a service type, a location, and the like, and filtering of log information is performed by the graph generation engine 220 according to the designation. These steps are all performed offline. The graphs generated by the graph generation engine 220 are provided to the user terminal 100 via the user interface 210, respectively.

[Second Embodiment]
In the first embodiment described above, log information is accumulated in the log DB 260 in advance, and the case of applying the rule grasp offline is considered. However, an event pre-registered online (for example, a failure) is used as a trigger. In this embodiment, an online example is shown.

  FIG. 5 is a system configuration example (online use example) of the second embodiment of the present invention.

In the figure, the same components as those in FIG. The system shown in the figure has a configuration in which a failure monitoring device 400 of an external device and a trigger event DB 270 are added to the configuration of FIG. Failure monitoring unit 400, for example, upon detecting a failure of the monitored devices 310 _1, and stores the failure information to the trigger event DB 270. The graph generation engine 220 generates a graph with reference to the log group DB 240, the log template DB 250, and the log DB 260 of the first embodiment, triggered by the failure information stored in the trigger event DB 270, and the user interface 210. Present the graph to the user via

  Which template the log information sent from the log collection device 230 belongs to, which log template group belongs to the user's terminal 100 via the user interface 210 is grasped and monitored simultaneously. Device information is also grasped.

  In addition, the processing of the user interface 11, the file control unit 12, the graph generation unit 13, the user interface 210 of the example, and the graph generation engine 220 in the above embodiment is constructed as a program and used in a computer used as a log visualization device. It can be installed and executed, or distributed via a network.

  The present invention is not limited to the above-described embodiments, and various modifications and applications can be made within the scope of the claims.

10 Log Visualization Device 11 User Interface 12 File Control Unit 13 Graph Generation Unit 14 Log Group DB
15 Log template DB
16 Log DB
20 Input device 30 Display device 40 Log collection device 50 Monitoring target device 100 User 200 Log visualization device 210 User interface 220 Graph generation engine 230 Log collection device 240 Log group DB
250 Log template DB
260 Log DB
270 Trigger Event DB
300 Monitoring target device group 310 Monitoring target device

Claims (8)

A log visualization device for visualizing log information of multiple monitored devices,
Log information storage means for storing log information including device identification information for each device to be monitored and a monitoring device label given by an additional attribute of the device to be monitored;
A log that stores template information in which an important part extracted from the log information (hereinafter referred to as “log template”) is given a label (hereinafter referred to as “template label”) according to the usage status. Template storage means;
Log group storage means for storing a log group in which the template information is grouped based on the co-occurrence of log information corresponding to the template information ;
And said log information before Symbol log information storage unit, performs matching between the log template storage means or information of the log group storage unit, a log group, a log template label ordinate, the time of occurrence of the log information to the horizontal axis Graph generating means for generating a graph by plotting and outputting to a display device;
A log visualization apparatus characterized by comprising: The graph generation means includes:
On the vertical axis of the graph,
Mapping the log template and the log group so as to take a higher value as the frequency of the log group is lower.
Mapping so as to take a higher value as the appearance order of the log template and the log group is slower,
Mapping the log template and the log group with a low periodicity value so as to take a high value,
Mapping so that the log template and log group that are similar are close
Define and map the order relationship between the log template and the log group in advance.
The log visualization apparatus according to claim 1, further comprising means for drawing the graph by combining at least one or a plurality of the graphs. The graph generation means includes:
The log visualization device according to claim 1, further comprising means for dividing and displaying the graph for each device type of the monitored device. The graph generation means includes:
2. The display device according to claim 1, further comprising means for changing a display of the symbol used for displaying the graph in accordance with the number of log templates generated in the log group or the number of log templates and log groups frequently generated in the same time. Log visualization device. The graph generation means includes:
The log visualization apparatus according to claim 1, further comprising means for changing the shape, color, and size of the symbol for each device identification information. A log visualization method for visualizing log information of a plurality of monitored devices,
Log information storage means for storing log information including device identification information for each device to be monitored and a monitoring device label given by an additional attribute of the device to be monitored;
A log that stores template information in which an important part extracted from the log information (hereinafter referred to as “log template”) is given a label (hereinafter referred to as “template label”) according to the usage status. Template storage means;
An apparatus comprising: a log group storage means for storing the log group to which the template information is grouped on the basis of the co-occurrence of log information corresponding to the template information, the graph generating unit, a,
The graph generation unit, and the log information before Symbol log information storage unit, performs matching between the log template storage means or information of the log group storage unit, a log group, a log template label ordinate, the log information graph generating steps to generate a graph by plotting the generation time on the horizontal axis, and output to the display device,
A log visualization method characterized by: In the graph generation step,
On the vertical axis of the graph,
Mapping the log template and the log group so as to take a higher value as the frequency of the log group is lower.
Mapping so as to take a higher value as the appearance order of the log template and the log group is slower,
Mapping the log template and the log group with a low periodicity value so as to take a high value,
Mapping so that the log template and log group that are similar are close
Define and map the order relationship between the log template and the log group in advance.
The log visualization method according to claim 6, wherein the graph is drawn by combining at least one of or a plurality of the graphs. Computer
The log visualization program for functioning as each means of the log visualization apparatus of any one of Claims 1 thru | or 5.

Images