JP6280089B2 - Whitelist creation device, whitelist creation method, and whitelist creation program - Google Patents

Description

  The present invention relates to a white list creation device, a white list creation method, and a white list creation program.

  Conventionally, a black list method and a white list method are known as methods for monitoring abnormal logs. In the black list method, a log that matches a keyword included in the black list is determined as an abnormal log. On the other hand, in the white list method, a log that does not match a keyword included in the white list is determined as an abnormal log.

  For example, Zabbix and Hinemos, which are operation monitoring systems, generally use the black list method. On the other hand, with Fluent or tenshi using the rewrite-tag-filter plug-in, it is possible to monitor abnormal logs using the whitelist method.

  The black list method is characterized by high detection accuracy for known abnormal logs, but low detection accuracy for unknown abnormal logs. On the other hand, since the white list is created from a log determined to be normal, the white list method has high detection accuracy for an unknown abnormal log. Therefore, the white list method may be used for the purpose of detecting an unknown log that could not be detected by the black list method, for example.

  Further, in the black list method, it is only necessary to determine that a log including a portion that matches a keyword is abnormal, so that the number of filtering rules included in the black list can be reduced. On the other hand, in the white list method, it is required to detect abnormal logs without omission and to reduce normal logs included in the notification to the user. For this reason, it is necessary to include a larger amount of filtering rules in the white list than in the black list. If the filtering rules included in the white list are insufficient, a large amount of detection notifications are generated to the user, and there is a possibility that a truly necessary notification may not be recognized by the user.

“Zabbix” [online], [October 2, 2015 search], Internet (http://www.zabbix.com/) “Hinemos” [online], [October 2, 2015 search], Internet (http://www.hinemos.info/) “Fluentd” [online], [October 2, 2015 search], Internet (http://www.fluentd.org/) “Tenshi” [online], [October 2, 2015 search], Internet (http://inversepath.com/tenshi.html)

  However, the conventional white list method has a problem that a white list including sufficient filtering rules cannot be efficiently created.

  For example, when creating a white list including a large amount of filtering rules using regular expressions, at present, the white list must be created manually based on the log. For this reason, the conventional technique cannot efficiently create a white list including a sufficient filtering rule using a regular expression.

  In addition, by adding the log to the white list as it is, the amount of work for creating the white list can be reduced, but in that case, the filtering rules included in the white list may be insufficient. For this reason, adding a log as it is may not create a whitelist including sufficient filtering rules.

  The whitelist creation device of the present invention includes an input unit that receives normal log input, an extraction unit that extracts a portion that matches a preset extraction condition from the log, and a portion extracted by the extraction unit. , Using a whitelist that is a set of rules for filtering normal logs, the conversion unit for converting into a regular expression corresponding to the extraction condition, and the log in which the extracted part is converted by the conversion unit And an output unit that outputs to a log monitoring device that monitors the log.

  In addition, the whitelist creation method of the present invention is extracted by the input step of accepting normal log input, the extraction step of extracting a portion that matches a preset extraction condition from the log, and the extraction step. A whitelist that is a set of rules for filtering normal logs, a conversion step of converting a portion into a regular expression corresponding to the extraction condition, and the log into which the extracted portion has been converted by the conversion step And an output step of outputting to a device for monitoring the log using.

  According to the present invention, a white list including sufficient filtering rules can be efficiently created.

FIG. 1 is a diagram illustrating an example of a configuration of a log monitoring system according to the first embodiment. FIG. 2 is a diagram illustrating an example of a configuration of a conversion unit of the white list creation apparatus according to the first embodiment. FIG. 3 is a diagram illustrating an example of an extraction condition and a conversion method. FIG. 4 is a diagram illustrating an example of a log before conversion and after conversion. FIG. 5 is a sequence diagram illustrating an example of processing of the log monitoring system according to the first embodiment. FIG. 6 is a flowchart illustrating an example of processing of the white list creation apparatus according to the first embodiment. FIG. 7 is a diagram illustrating an example of a computer in which a whitelist creation apparatus is realized by executing a program.

  Embodiments of a white list creation device, a white list creation method, and a white list creation program according to the present application will be described below in detail with reference to the drawings. In addition, this invention is not limited by this embodiment.

[First Embodiment]
First, the configuration, processing flow, and effects of the log monitoring system and the white list creation device according to the first embodiment will be described.

[Log Monitoring System Configuration]
First, the configuration of the log monitoring system will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of a configuration of a log monitoring system according to the first embodiment. As shown in FIG. 1, the log monitoring system 1 includes a white list creation device 10 and a log monitoring device 20.

  The white list creation device 10 creates a white list that is a set of rules for filtering normal logs based on a log or the like input from a user. The white list creation device 10 includes an input unit 11, an extraction unit 12, a conversion unit 13, an output unit 14, and a storage unit 15.

  The input unit 11 receives log input. For example, a log that the user has determined to be normal is input to the input unit 11. Here, for example, a log that the user has determined to be abnormal is not input to the input unit 11. The input unit 11 receives a log input by reading a log file from a user terminal connected via a network, for example.

  The extraction unit 12 extracts a part that matches a preset extraction condition from the log input to the input unit 11. The extraction condition is represented by a regular expression, for example.

  The conversion unit 13 converts the part extracted by the extraction unit 12 by a conversion method corresponding to the extraction condition. For example, the conversion unit 13 converts the part extracted by the extraction unit 12 into a regular expression corresponding to the extraction condition. Here, it is assumed that the storage unit 15 stores the extraction condition and the conversion method corresponding to the extraction condition.

  For example, the extraction unit 12 extracts a portion that matches the extraction condition of being a decimal number or a hexadecimal number. At this time, the conversion unit 13 converts the portion into a regular expression corresponding to a decimal number or a hexadecimal number. Further, for example, the extraction unit 12 extracts a portion that matches the extraction condition that the blanks are continuous. At this time, the conversion unit 13 converts the portion into a regular expression corresponding to a sequence of a plurality of blanks. Further, for example, the deletion unit 131 of the conversion unit 13 deletes a part representing the date and time of the beginning of the log and the host name from the log. The output unit 14 outputs a log from which the part has been deleted by the deletion unit 131.

  The output unit 14 outputs the log whose part has been converted by the conversion unit 13 to the log monitoring device 20.

  The storage unit 15 stores an extraction condition used by the extraction unit 12 and a conversion method corresponding to the extraction condition used by the conversion unit 13. Details of the extraction conditions and the conversion method stored in the storage unit 15 will be described later.

  Further, the log monitoring device 20 monitors logs using the white list created by the white list creation device 10. The log monitoring device 20 includes a setting unit 21, a storage unit 22, and a monitoring unit 23.

  The setting unit 21 causes the storage unit 22 to store the converted log acquired from the output unit 14 of the white list creation apparatus 10 as a white list. The monitoring unit 23 uses the white list stored in the storage unit 22 to monitor logs as real time or batch processing.

  At this time, the monitoring unit 23 determines that a log that matches the regular expression described in the white list is a normal log. Further, the monitoring unit 23 determines that a log that does not match the regular expression described in the white list is an abnormal log. The monitoring unit 23 may further perform monitoring using a black list as well as monitoring using a white list.

[Configuration of whitelist creation device]
Next, the configuration of the conversion unit of the white list creation device will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a configuration of a conversion unit of the white list creation apparatus according to the first embodiment. The white list creation device 10 converts the extracted log into a regular expression. Here, if the log is changed to a regular expression “. *”, All the logs are included in the regular expression, and an abnormal log cannot be detected. For this reason, avoid using a regular expression unless the log format is clear in advance and it is known that there is no problem even if you combine multiple logs using regular expressions.

  On the other hand, in the case of a numerical value and a space character, it does not have a meaning by itself, and if the content of the log is different due to different numerical value parts, the message content represented by the character string is also considered to change. For example, it is conceivable that the error code represented by a numerical value varies depending on the log. In this case, when the error code changes, the content of the error displayed in the log also changes. For this reason, it is considered that there is no problem even if numerical values and space characters are converted into regular expressions.

  As illustrated in FIG. 2, the conversion unit 13 includes a deletion unit 131, an escape processing unit 132, a specifying unit conversion unit 133, a number conversion unit 134, an address conversion unit 135, a space conversion unit 136, and a date / time conversion unit 137. Each unit of the conversion unit 13 performs processing according to the conversion method stored in the storage unit 15.

  Here, the extraction conditions and the conversion method stored in the storage unit 15 will be described with reference to FIG. As illustrated in FIG. 3, the storage unit 15 stores extraction conditions and a conversion method. Some extraction conditions can specify the position in advance according to the setting of the device that outputs the log, such as “date and time of the sentence head and host name”. Some extraction conditions, such as “predetermined ID”, are known to have no problem even if they are converted into regular expressions in advance. In addition, the extraction condition may be a character string that matches a condition represented by a regular expression, such as “date and time”, “IP address of IPv4 host”, or “hexadecimal number”.

For example, the extraction condition “hexadecimal number” can be specifically expressed as (1) to (3) below. In hexadecimal numbers, 0 to 9, a to f, and A to F are used as numerical values.
(1) Character string starting with "0x", "tx", "tap", "req-", "id =", and consisting of a numerical value and [a-fA-F] ("-" delimiter is allowed) .
(2) A character string composed of one or more numeric values and [a-fA-F], sandwiched between delimiters (such as "-").
(3) A character string consisting of [a-fA-F] delimited by “-”, which is a character string before and after matching (1) or (2).

  For example, the conversion method corresponding to the extraction condition “date and time of the sentence and host name” is “deletion”. In this case, the deletion unit 131 deletes a part that matches the extraction condition. The conversion method corresponding to the extraction condition “any character of \, |, (,), {,}, [,]” is “add“ \ ”before the character”. In this case, the escape processing unit 132 adds “\” before the character that matches the extraction condition.

  For example, the conversion method corresponding to the extraction condition “predetermined ID” is “convert to regular expression“ \ w + ””. In this case, the identifying unit conversion unit 133 converts the portion that matches the extraction condition into “\ w +”. It is assumed that the predetermined ID is expressed in advance in single-byte alphanumeric characters and appears to appear after a specific character string.

  For example, the conversion method corresponding to the extraction condition “hexadecimal” is “convert to regular expression“ [\ da-fA-F] + ””. In this case, the number conversion unit 134 converts the part that matches the extraction condition into “[\ da-fA-F] +”.

  For example, the conversion method corresponding to the extraction condition “IP address of IPv4 host” is “convert to regular expression“ (\ d {1,3} \.) {3} \ d {1,3} ””. . In this case, the address conversion unit 135 converts the part that matches the extraction condition into “(\ d {1,3} \.) {3} \ d {1,3}”.

  For example, the conversion method corresponding to the extraction condition “continuous white space” is “convert to regular expression“ \ s + ””. In this case, the blank conversion unit 136 converts the portion that matches the extraction condition into “\ s +”.

  For example, the conversion method corresponding to the extraction condition “date and time” is “regular expression” (Jan | Feb | Mar | Apr | May | Jun | Jul | Aug | Sep | Oct | Nov | Dec) \ \ d {1, 2} \ s [\ d \] \ d: \ d {2}: \ d {2} ". In this case, the date and time conversion unit 137 selects the part that matches the extraction condition as “(Jan | Feb | Mar | Apr | May | Jun | Jul | Aug | Sep | Oct | Nov | Dec) \ \ d {1,2} \ s [\ d \] \ d: \ d {2}: \ d {2} ".

  In addition, the extraction part 12 and the conversion part 13 perform extraction and conversion in order of NO of FIG. Therefore, for example, the date and time part deleted in NO1 is not subject to conversion in NO4. For example, in the case of a log output from a system that uses a space character as a delimiter, a space character escape process is required.

  Since the IP address and the MAC address are often expressed using decimal numbers or hexadecimal numbers, NO6 to NO8 by the address conversion unit 135 are arranged before the conversion of NO9 to NO11 by the number conversion unit 134. Thus, it is possible to prevent the address from being converted as a number.

  Next, log conversion will be described with a specific example. FIG. 4 is a diagram illustrating an example of a log before conversion and after conversion. The part 1a in FIG. 4 represents the date and host name and the host name at the beginning of the sentence, and is deleted because it matches the extraction condition of NO1 in FIG. The part 2a in FIG. 4 is a message ID that appears after “sendmail”, and is converted into a regular expression as shown in 2b that matches the extraction condition of NO2 in FIG.

  Further, since the portions 3a and 5a in FIG. 4 match the extraction conditions of NO3 and NO5 in FIG. 3, respectively, escape processing is performed as shown in 3b and 5b. Further, since the portions 4a and 6a to 12a in FIG. 4 match the extraction conditions of NO4 and NO6 to NO12 in FIG. 3, respectively, they are converted into regular expressions as shown in 4b and 6b to 12b.

[Process of First Embodiment]
The process of the log monitoring system 1 will be described with reference to FIG. FIG. 5 is a sequence diagram illustrating an example of processing of the log monitoring system according to the first embodiment. As shown in FIG. 5, first, the user inputs a log determined to be normal to the white list creation device 10 (step S101).

  The white list creation apparatus 10 accepts log input (step S102). Next, the white list creation apparatus 10 extracts a portion that matches a predetermined extraction condition from the log, and performs conversion by a predetermined conversion method (step S103). Then, the whitelist creation apparatus 10 outputs the log that has been extracted and converted to the log monitoring apparatus 20 (step S104).

  The log monitoring device 20 sets the log acquired from the white list creation device 10 in the white list (step S105). And the log monitoring apparatus 20 monitors a log using a white list (step S106).

  Further, the log monitoring device 20 feeds back the log to the white list creation device 10 (step S107). In this way, the log monitoring system 1 performs learning by operating the white list and feeding back, for example, a log that is erroneously determined to be an abnormal log to the white list creating apparatus 10 again, and this is repeated. The list may be expanded.

  At this time, the input unit 11 receives an input of a log that is determined as a normal log by the log monitoring device 20. Thereby, an unknown normal log is classified as an abnormal log when it first appears, but is classified as a normal log after learning by feedback.

  Here, the processing of the white list creation apparatus 10 will be described with reference to FIG. FIG. 6 is a flowchart illustrating an example of processing of the white list creation apparatus according to the first embodiment. First, as illustrated in FIG. 6, the input unit 11 of the whitelist creation apparatus 10 receives log input (step S <b> 201).

  Next, the extraction unit 12 of the whitelist creation apparatus 10 extracts a portion that matches the extraction condition stored in the storage unit 15 from the log (step S202). Then, the conversion unit 13 of the whitelist creation apparatus 10 converts the log extracted according to the conversion method corresponding to the extraction condition (step S203).

  At this time, if there is an extraction condition that has not been extracted (step S204, Yes), the process returns to step S202, and extraction is performed for the extraction condition that has not been extracted. If there is no extraction condition that has not been extracted (No at Step S204), the converted log is output to the log monitoring device 20 (Step S205), and the process is terminated.

[Effect of the first embodiment]
The accuracy of log monitoring can be expressed by the following recall and precision.
Reproducibility = (Number of truly abnormal logs among abnormally determined logs) / (Total number of truly abnormal logs)
Compliance rate = (the number of logs that are determined to be abnormal among the logs that are determined to be abnormal) / (the total number of logs that are determined to be abnormal)

  In general, the black list method describes the characteristics of a known abnormality log with a regular expression. Therefore, although the relevance rate is close to 100%, the reproducibility is not so high, and there is a tendency to be weak in detecting an unknown log. On the other hand, the purpose of the white list method is to detect an abnormal log leaked by the black list method, particularly an unknown abnormal log, in order to improve the problems of the black list method.

  For this reason, in the white list method, the first goal is to achieve a recall rate of 100%, and it is also necessary to increase the matching rate. This requires a whitelist that contains sufficient filtering rules.

  At this time, it is difficult to create a white list including sufficient filtering rules by manually creating a white list or adding a log as it is to the white list as in the past.

  Therefore, the input unit 11 of the whitelist creation apparatus 10 first receives a normal log input. Then, the extraction unit 12 extracts a portion that matches a preset extraction condition from the log. Here, the conversion unit 13 converts the portion extracted by the extraction unit 12 into a regular expression corresponding to the extraction condition. Thereafter, the output unit 14 outputs the log whose portion has been converted by the conversion unit 13 to a device that monitors the log using a white list that is a set of rules for filtering normal logs.

  As described above, since the white list creation apparatus 10 can automatically create a white list represented by a regular expression, a white list including a sufficient filtering rule can be efficiently created. Furthermore, by using a white list including sufficient filtering rules created by the white list creation device 10, it is possible to improve the conformance rate in log monitoring.

  Further, for example, the extraction unit 12 extracts a portion that matches the extraction condition of being a decimal number or a hexadecimal number. At this time, the conversion unit 13 converts the portion into a regular expression corresponding to a decimal number or a hexadecimal number.

  Further, for example, the extraction unit 12 extracts a portion that matches the extraction condition that the blanks are continuous. At this time, the conversion unit 13 converts the portion into a regular expression corresponding to a sequence of a plurality of blanks.

  Further, for example, the deletion unit 131 of the conversion unit 13 deletes a part representing the date and time of the beginning of the log and the host name from the log. The output unit 14 outputs a log from which the part has been deleted by the deletion unit 131.

  In this way, the white list creation device 10 can create a white list from various types of logs depending on the settings of the extraction unit 12 and the conversion unit 13.

  In addition, when the log was monitored using the white list created by the white list creation device 10, the reproduction rate for the log including the character string was able to be 100%. The relevance ratio at that time largely depends on log characteristics and log data for learning. For example, when a new normal log is unlikely to appear and there is no large fluctuation in the normal log, the relevance rate is high and can be close to 100%, so the present invention is useful for detecting an unknown abnormal log.

  Furthermore, in combination with blacklist determination, logs that are known to have problems in advance can be separated by blacklists, and whitelists can be applied to the remaining logs to achieve higher accuracy. .

  Furthermore, the input unit 11 of the whitelist creation apparatus 10 may accept an input of a log that is determined as a normal log by the log monitoring apparatus 20. In this case, the white list creation device 10 can create a white list that can further improve the relevance ratio in log monitoring by receiving and learning the monitoring result of the log monitoring device 20 as feedback.

[System configuration, etc.]
Further, each component of each illustrated apparatus is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. Further, all or any part of each processing function performed in each device is realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or hardware by wired logic. Can be realized as

  In addition, among the processes described in the present embodiment, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or a part can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-described document and drawings can be arbitrarily changed unless otherwise specified.

[program]
As an embodiment, the white list creation device 10 can be implemented by installing a white list creation program for executing the above monitoring as package software or online software on a desired computer. For example, by causing the information processing apparatus to execute the above white list creation program, the information processing apparatus can function as the white list creation apparatus 10. The information processing apparatus referred to here includes a desktop or notebook personal computer. In addition, the information processing apparatus includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDA (Personal Digital Assistants).

  The white list creation device 10 can also be implemented as a server device that uses a terminal device used by a user as a client and provides the client with the above-described service related to white list creation. For example, the white list creation device 10 is implemented as a server device that provides a white list creation service that takes a log before conversion as input and outputs a log after conversion as output. In this case, the white list creation device 10 may be implemented as a Web server, or may be implemented as a cloud that provides the above-described service related to white list creation by outsourcing.

  FIG. 7 is a diagram illustrating an example of a computer in which a whitelist creation apparatus is realized by executing a program. The computer 1000 includes a memory 1010 and a CPU 1020, for example. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These units are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to the display 1130, for example.

  The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the whitelist creation apparatus 10 is implemented as a program module 1093 in which a code executable by a computer is described. The program module 1093 is stored in the hard disk drive 1090, for example. For example, a program module 1093 for executing the same processing as the functional configuration in the whitelist creation apparatus 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

  The setting data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1090 to the RAM 1012 and executes them as necessary.

  The program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read out by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and the program data 1094 may be read by the CPU 1020 from another computer via the network interface 1070.

DESCRIPTION OF SYMBOLS 1 Log monitoring system 10 White list production apparatus 20 Log monitoring apparatus 11 Input part 12 Extraction part 13 Conversion part 14 Output part 15,22 Storage part 21 Setting part 23 Monitoring part 131 Deletion part 132 Escape processing part 133 Specific part conversion part 134 Number Conversion unit 135 Address conversion unit 136 Blank conversion unit 137 Date and time conversion unit

Claims (7)

An input unit that accepts normal log input;
An extraction unit that extracts a part that matches a preset extraction condition from the log;
A conversion unit that converts the part extracted by the extraction unit into a regular expression corresponding to the extraction condition;
An output unit that outputs the log obtained by converting the extracted part by the conversion unit to a log monitoring device that monitors a log using a white list that is a set of rules for filtering normal logs;
A white list creation device characterized by comprising: The extraction unit extracts a portion that matches an extraction condition of being a decimal number or a hexadecimal number,
The whitelist creation apparatus according to claim 1, wherein the conversion unit converts the part extracted by the extraction unit into a regular expression representing a character string used as a numerical value or a numerical value in hexadecimal. The extraction unit extracts a portion that matches an extraction condition that a space is continuous,
The whitelist creation device according to claim 1 or 2, wherein the conversion unit converts the portion extracted by the extraction unit into a regular expression representing one or more blanks. The log further comprises a deletion unit that deletes a part representing the date and host name of the beginning of the log,
4. The output unit according to claim 1, wherein the output unit outputs, to the log monitoring device, the log in which a part representing a date and a host name of the log is deleted by the deletion unit. 5. The white list creation device described in 1. The input unit further accepts an input of a log determined as a normal log by the log monitoring device;
5. The whitelist creation according to claim 1, wherein the extraction unit extracts a portion that matches a predetermined condition from a log determined as a normal log by the log monitoring device. apparatus. An input process for accepting normal log input;
An extraction step for extracting a part that matches a preset extraction condition from the log;
A conversion step of converting the part extracted by the extraction step into a regular expression corresponding to the extraction condition;
An output step of outputting the log obtained by converting the extracted part by the conversion step to an apparatus for monitoring the log using a white list which is a set of rules for filtering normal logs;
A white list creation method characterized by including   A whitelist creation program for causing a computer to function as the whitelist creation device according to any one of claims 1 to 5.

Images