TWI548985B - Identification system of network log format and its method - Google Patents

Description

Network log format identification system and method thereof

The invention belongs to a method for identifying a web log format.

A network log is an event or operational record generated by a device on the network. Different uses or different types of devices will generate log files in different formats. For log files of different formats, there is usually a corresponding set of log parsers for parsing the contents of the log.

When processing log files of different formats at the same time, the methods currently used are mainly the following:

1. Specify the format of the log file and the corresponding log parser in a manually specified manner.

2. The specific keyword contained in the log file content, and the log file format and the corresponding log parser are determined by keywords.

3. The specific format of the specific field of the log file content, in a regular expression to capture a specific field within the log file.

When a new log format is added, the manually specified method requires the cost of human intervention, and the decision of the keyword or the formal representation also needs to be generated at a certain cost to generate a corresponding keyword and a regular expression. With the advancement of this data analysis technology, more and more applications are to achieve solutions by integrating logs from different devices. Maintaining different log formats in case of such demand The cost will also increase relatively.

In view of the shortcomings derived from the above-mentioned conventional methods, the inventor of the present invention has improved and innovated, and after years of painstaking research, he finally succeeded in researching and developing the identification system and method of the weblog format.

When processing network log data, you need to first identify the format of the log before you can use the corresponding log parser. The purpose of the network log format identification system and its method is to tell the log parser the log format to be processed in an automated process without manual designation.

In order to achieve the above object, a log format recognition system includes the following two modules: a log format recognition module, which is used to identify an original log file of an unknown format. After entering the original log file in an unknown format, a partial record is taken for the original log file. The record for the sample is converted into a string of the biological sequence according to the log format feature coding table. The log format feature coding table has corresponding symbol for the delimiter symbol and the punctuation symbol, and the part of the column content is processed according to the data type of the content, and one symbol represents a data type.

The converted sequence is searched through the biological sequence alignment to search the log format template database to find the most similar log format. If the sampled records find the same log format, a set of log format names is output; if there are multiple groups, the output log format name list is sorted according to the score.

The log format template establishes a module, which is used to establish a template database of the log format and adjust the weight of the scoring mechanism in the sequence alignment. Collect raw log files in a known format. Record each record in the file according to the log grid The character encoding table is converted into a string of the biological sequence, and is stored as a pair of the log format name corresponding to the sequence string, which is called a log format template. Through the filtering process, the qualified template is saved to the log format template database. In the log format template database, the same log format may correspond to multiple sequence strings, and one log format may be stored in multiple log format templates. After the log format template database is established, the scoring matrix of the sequence alignment is adjusted according to the sequence distribution of the log grid template in the sequence database.

In order to achieve the above object, the method for identifying a log format includes: first establishing a log sequence template database; the user collects the original log file in a known format, and establishes a log format template database through the log format template. Then, through the content of the log format template database, the scoring matrix of the sequence alignment sequence is added. If a new log format is added later, the module update log format template database and the sequence alignment scoring matrix are also established through the log format template. After the log format template database is created, the user inputs the original log file of the unknown log format; the log format identification module first samples the contents of the original log file, and encodes the sampled record into a sequence of biological sequences; The string is searched for the sequence of the log format template database to find the most similar sequence; the corresponding log format name is determined as the log format of the original log file as an output.

When the invention is used, it is necessary to first establish a log format template serial Library. The user collects the original log file in a known format, and first creates a log format template database through the log format template. The score matrix of the sequence alignment is then adjusted through the contents of the log format template database. If a new log format is added later, the module update library format database and sequence alignment matrix are created through the log format template.

After setting up the log format template database, the user enters The original log file for the unknown log format. The log format identification module first samples the contents of the original log file and encodes the sampled records into a sequence of biological sequences. Then, in this string, the sequence of the log format template database is searched to find the most similar sequence, and the corresponding log format name is determined as the log format of the original log file as an output.

The invention provides a network log format identification system and a method thereof, which have the following advantages when compared with other conventional technologies:

The present invention provides a mechanism for converting a log into a biological sequence string, which can convert a log into a sequence of a biological sequence according to a log format feature encoding table, store the pairing of the sequence string corresponding to the log format name, and establish a log format template. For future log comparison use.

2. The invention utilizes the sequence alignment technology of biological information, can establish a fast index of all sequences, generate a log database of various templates, and establish a fast index to effectively improve the efficiency of the comparison.

3. The present invention provides a log format template establishment mechanism. The user can specify the log format name of the original log file, and the log format feature can be built into the log format template database to provide subsequent query usage, which will be followed. The growth of the log format database, the recognition ability and accuracy for different log formats will also increase.

4. The present invention provides a log identification mechanism for original days of unknown formats The file can automatically and quickly identify the format of the log, which can minimize the maintenance cost of the manual intervention decision log format.

A‧‧‧ original log file of known format

B‧‧‧Original log file in unknown format

100‧‧‧ computer system

110‧‧‧Log format template creation module

111, 122, 520‧‧‧ log coding module

112‧‧‧Sample production module

113, 542‧‧ ‧ weight adjustment module

120‧‧‧Log format recognition module

121‧‧‧File sampling module

123‧‧‧Sequence comparison module

130, 541‧‧‧ log format template database

140, 550‧‧‧ log format name

510‧‧‧ Original log records

530‧‧‧Coded sequence

540‧‧‧Sequence comparison module

S601~S605‧‧‧Log format identification process

S701~S704‧‧‧Log format template database establishment process

The detailed description of the present invention and the accompanying drawings will be further understood, and the technical contents of the present invention and the functions thereof can be further understood. FIG. 1 is a system architecture of the identification system of the network log format and the method thereof. 2 is a schematic diagram of a log file feature side code table conversion original log record of the network log format recognition system and method thereof; FIG. 3 is a BLAST preset of the network log format recognition system and method thereof according to the present invention; The scoring matrix BLOSUM 62; FIG. 4 is a comparison diagram of each symbol and symbol of the log format template database of the identification system and method of the network log format of the present invention; FIG. 5 is a recognition system of the network log format of the present invention; The log format template of the method and the relationship diagram of the internal components of the log format identification module; FIG. 6 is a flow chart for identifying the log file format identification system and the method thereof; FIG. 7 is a network diagram of the present invention; A flow chart of the log format identification system and the log format template database of the method.

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Hereinafter, the present invention will be further described with reference to the accompanying drawings: Please refer to FIG. 1 , which is a schematic diagram of a system architecture of a network log format identification system and a method thereof, including two main modules disposed under the computer system 100: a log format template creation module 110 and a log format recognition. Module 120. The log format template establishing module 110 is configured to collect training materials to generate a log format template database for use by the log format identification module; the log format identification module 120 is mainly used for collecting the network log, and the unknown log format is The file identifies the log format name to aid in the analysis and processing of subsequent data.

As shown in FIG. 1, the object of the present invention is to provide a log format recognition system, which can automatically and quickly establish a log format template and accurately identify the format of the log. The user collects the original log file A of the known format as the training data, and creates the log encoding module 111 in the module 110 through the log format template, and records each record in the file according to the log format feature coding table (Table 1 below). Converting into a string of the biological sequence, and then storing the string into a pair of the sequence string corresponding to the log format name through the template generating module 112, creating a log format template database 130, and finally passing the weight adjustment module 113, according to the log The content distribution of the format template database 130 adjusts the weight of the scoring matrix of the sequence alignment. In the log format template database 30, the same log format may correspond to multiple sequence strings, and one log format may be stored in multiple log format templates. If a new log format is added later, the module 10 is updated by the log format template to update the log format template database 30 and the score matrix of the sequence alignment.

When collecting web logs, when the user enters the original format of the unknown format After the log file B is started, the file sampling module 121 in the log format identification module 120 samples the content of the original log file, and then converts the sampled record into a biological sequence according to the log format feature coding table through the log encoding module 122. String, the log format feature code table has a corresponding representative symbol for the delimiter symbol and the punctuation symbol, and the part of the column content is processed according to the data type of the content, and a symbol represents a data type and is converted into a biological sequence. After the string is followed, the sequence comparison module 123 is used to search the sequence of the log format template database 130 to find the most similar sequence, and the corresponding log format name 140 is determined as the original log file. The log format is output. If the sampled records find the same log format, a set of log format names is output. If there are multiple groups, the output log format name list is sorted according to the score.

The specific log process is described as follows: The user collects the original log file A in a known format as the training data, and builds the module 10 through the log format template to construct the log format template database 30.

The original log file A of the known format is converted into a sequence of biological sequences according to the log format feature coding table (see Table 1) through the log coding module 111 in the log format template creation module 110. The relevant steps are as follows:

1. For the content enclosed by the punctuation mark ("), delete the part that contains the separator symbol and the punctuation mark.

2. Scanning from the first character of the original log record to the last character, the corresponding separator and punctuation marks are replaced by the corresponding representative symbols. If the separation or punctuation of the table is not encoded for the log format feature, the string is read until the next character is a delimiter or punctuation. Then judge the data type of the string, if it is a number, it is represented by I, and if it is not a number, it is represented by S, otherwise it is represented by X.

Please refer to FIG. 2, which is an example of the log file feature side code table conversion original log record of the network log format identification system and the method thereof.

The string converted into the biological sequence is stored by the template generation module into the pairing of the sequence string corresponding to the log format name, and the log format template is generated, and then stored in the log format template database.

As an example of the log format template name in Table 2 below, the module saves the sequence of different log formats into the format of Table 2, filters out the duplicate names and sequences, and then creates a fast index through BLAST to generate a log format template database 30.

In the log format template database 30, each log format contains more than one representative sequence, and the weight adjustment module 13 is mainly used to adjust the scoring matrix used by BLAST to perform multiple sequence alignment on all sequences in the same log format. Please refer to FIG. 4, through the multiple sequence comparison, the comparison between each symbol and the symbol of the log format template database can be obtained, and a scoring matrix can be calculated by the following formula: Sij=(1/ λ ) Log(Pij/Qi*Qj+0.5)

Where Pij represents the probability that symbol i and symbol j are ranked on the same line in multiple sequence alignments, Qi and Qj represent the probability that the symbols i and j appear in the background of any sequence in the log format template database, λ is a factor Make sure that the elements in the scoring matrix are integers, and the constant 0.5 is used to adjust the matrix elements to avoid the chance of zero. This module performs multiple sequence alignment on the sequence of the same log format in the log format sample database through the biological sequence tool ClusterW. Obtain the probability of inter-symbol comparison, and then calculate the scoring matrix through the above formula; if a new log format is added later, it is also to establish a module update log format template database and a sequence alignment scoring matrix through the log format template.

In the context of collecting network logs, after the user inputs the original log file in an unknown format, the log format identification module can effectively identify the log format name.

When the user inputs the original log file in an unknown format, the file sampling module will use part of the record to represent the contents of the entire original file, and the sampling effect is achieved by taking a few records to take a record.

After the sampling result is obtained, the log code module is used to convert the sampled record into a sequence of the biological sequence according to the log format feature coding table, and then through the sequence comparison module, for searching the log format template database to find the most similar The sequence and output results, the sequence alignment module uses the Basic Local Alignment Search Tool (BLAST) to search for the most similar sequence in the log format template database 30. The Score Matrix used by BLAST can be adjusted through the sequence distribution of the log format template database. See Figure 3 BLAST's default scoring matrix BLOSUM 62.

After finding the most similar sequence, the corresponding log format name The 40 is determined to be the log format of the original log file as the output. If the sampled records find the same log format, a set of log format names are output; if there are multiple groups, the output log format name list is sorted according to the score.

Please refer to Figure 5 to create a module and date for the log format template. The sequence format recognition module internal component relationship diagram, the sequence comparison module 540 will query the coded sample sequence 530 to the log format template database 541, and the most similar sequence template format name found will be regarded as a sequence comparison mode. The output of the group is the log format name 550. In addition, the log format template database 541 is also used as an input to the weight adjustment module 542, and the generated scoring matrix affects the result of the alignment of the sequence comparison module 540.

Please refer to FIG. 6 , which is a flow chart for identifying a log format according to the present invention. The flow is as follows: S601 inputs a log file of an unknown format; S602 samples a log file record, and encodes a format feature in the record into a sequence for biometric information; S603 compares the created log format database to find the most similar log format name; S604 outputs the log format name of the log file if the comparison is successful; if the comparison fails in S605, the output cannot be determined.

Please refer to the figure, which is a flowchart for establishing a log format template database of the present invention. The flow is as follows: S701 inputs a log file of a known format; S702 takes out each record of the log file, and encodes the format feature in the record into biological information. The sequence used, generated into a log format name The log format template paired with the sequence; S703 filters out the duplicate log format template to generate a log format template database; S704 uses the sequence distribution of the log format template database to recalculate the scoring weight of the sequence alignment.

The detailed description of the present invention is intended to be illustrative of a preferred embodiment of the invention, and is not intended to limit the scope of the invention. The patent scope of this case.

To sum up, this case is not only innovative in terms of technical thinking, but also has many of the above-mentioned functions that are not in the traditional methods of the past. It has fully complied with the statutory invention patent requirements of novelty and progressiveness, and applied for it according to law. Approved this invention patent application, in order to invent invention, to the sense of virtue.

A‧‧‧ original log file of known format

B‧‧‧Original log file in unknown format

100‧‧‧ computer system

110‧‧‧Log format template creation module

111, 122‧‧‧ log coding module

112‧‧‧Sample production module

113‧‧‧weight adjustment module

120‧‧‧Log format recognition module

121‧‧‧File sampling module

123‧‧‧Sequence comparison module

130‧‧‧Log format template database

140‧‧‧Log format name

Claims (9)

The invention relates to a network log format identification system, which mainly comprises: a log format identification module, which is used for collecting a network log, and identifies a log format name of an archive of an unknown log format to facilitate analysis and processing of subsequent data. And a log format template building module for collecting training data to generate a log format template database for use by the log format identification module. For example, the identification system of the network log format described in the first aspect of the patent application, wherein the log format template creation module further comprises: a log coding module, which is based on a log format feature coding table for each record in the file. Converting into a sequence of a biological sequence; a template generating module stores the string into a pair of log format name corresponding sequence strings, and creates a log format template database; and a weight adjustment module based on the log format template data The content distribution of the library adjusts the weight of the scoring matrix of the sequence alignment. The identification system of the network log format as described in claim 1, wherein the log format identification module further comprises: an archive sampling module, which is to sample the content of the original log file; the log format coding module is The record sampled by the file sampling module is converted into a sequence of the biological sequence according to the log format feature coding table; the sequence comparison module compares the sequence of the biological sequence with the sequence of the log format template database. Find the most similar sequence; and the log format template database, which stores the string of the biological sequence. The identification system of the network log format as described in claim 2, wherein the log format name is a log format determined to be the original log file as an output. The identification system of the network log format described in claim 3, wherein the log format feature coding table has a corresponding representative symbol for the delimiter symbol and the punctuation symbol, and the part of the column content is based on the content information. The type is treated by a symbol representing a data type and converted into a string of biological sequences. A method for identifying a network log format, comprising: first establishing a log file format sample database; the user collects the original log file in a known format, and establishes a log format template database through the log format template; The content adjustment sequence of the format template database adjusts the scoring matrix of the sequence. If a new log format is added later, the module is updated by the log format template to update the log format template database and the sequence alignment of the sequence alignment; After the log format template database, the user inputs the original log file of the unknown log format; the log format recognition module first samples the content of the original log file, and encodes the sampled record into a sequence of biological sequences; and then uses the string Search the sequence of the log format template database to find the most similar sequence; the corresponding log format name is determined as the log format of the original log file as the output. The method for identifying a weblog format as described in claim 6, wherein the sequence alignment determines the similarity of the sequence according to the adjusted weight scoring mechanism. The method for identifying a network log format as described in claim 6 of the patent application scope, wherein the log format identification includes a process: Enter a log file of unknown format; sample the log file records, encode the format features in the record into a sequence for biometrics; compare the established log format database to find the most similar log format name; If successful, the log format name of this log file is output; if the comparison fails, the output cannot be determined. The method for identifying a network log format as described in claim 6, wherein the log format template database has a process of establishing: inputting a log file of a known format; and extracting each record of the log file, recording The format feature is encoded into a sequence for bioinformatics, and a log format template that matches the log format name and the sequence is generated; the duplicate log format template is filtered out to generate a log format template database; and the sequence distribution of the log format template database is utilized. Recalculate the scoring weights of the sequence alignments.