KR20230166726A - Apparatus for processing log data and method thereof - Google Patents

Abstract

This disclosure discloses a log data management method and system. The log data management method according to the present disclosure is performed by a computing system, comprising: masking some of the tokens of log data including a plurality of log sequences; aligning the masked log sequences; , It may include clustering the log sequences using the alignment results of the log sequences.

Description

Log data management method and system{APPARATUS FOR PROCESSING LOG DATA AND METHOD THEREOF}

This disclosure relates to a log data management method and system. More specifically, it relates to a method and system for classifying similar patterns using given log data to form clusters and determining which cluster newly input log data corresponds to.

As computer systems and various automated manufacturing facilities operate, their operating status and processing status are recorded in the form of log data. The log data can be used in various fields such as checking for system abnormalities, extracting statistical information, and optimization work.

For this purpose, the log data must be first processed and preprocessed to extract the desired data. To perform most preprocessing tasks, separate software must be written to extract data for each type of log data, which requires a lot of time and effort.

However, log data that is not standardized can have various forms depending on the system, manufacturer, S/W version, etc., so the log format may vary depending on the user's settings. Because of this, there is a problem in that in order to process various types of log data, users must understand the format of the target log and individually implement data processing logic appropriate for it.

Accordingly, there is an emerging need for a method to improve the cost-inefficiency problem of the log data processing process and quickly perform log data preprocessing by clustering log data without manual operation by the user.

The technical task to be achieved through some embodiments of the present disclosure is to provide a method for generating a log data clustering model that is efficient in terms of computing resource usage and a method for classifying log data using the log data classification model.

Another technical task to be achieved through some embodiments of the present disclosure is to quickly perform cluster search using a linear algorithm without having to calculate the distance to all clusters when determining which type newly input log data belongs to. To provide a log data management method and system.

Another technical task to be achieved through some embodiments of the present disclosure is to save time and capacity by reducing the number of searches as much as deleted child nodes and securing memory when the log data management system performs clustering of new log data sequences. This provides cost savings.

Another technical task to be achieved through several embodiments of the present disclosure is to provide a log data management method and system that can quickly reflect the user's intention and process data in the form of a stream input in real time without delay. .

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

In order to solve the above technical problem, a log data management method according to an embodiment of the present disclosure includes masking some of the tokens of log data including a plurality of log sequences, and sorting the masked log sequences. It may include the step of clustering the log sequences using the sorting results of the log sequences.

The masking step may include identifying a masking target token for each of the plurality of log sequences and replacing the masking target token with a masking expression corresponding to the masking target token.

The step of replacing the masking target token with a masking expression corresponding to the masking target token includes determining a type of the masking target token and replacing the masking target token with a masking expression corresponding to the type of the masking target token. May include steps.

The log data management method may further include generating a clustering model using the clustering result.

The step of creating the clustering model includes creating a trie data structure with each token as a node, and each node may have a reference count for each cluster as an attribute value.

Generating the trie data structure may include determining a representative log sequence for each cluster and generating the trie data structure using the representative log sequences.

The step of generating the trie data structure includes merging two or more consecutive nodes having a single child node when there are two or more consecutive nodes having a single child node in the direction from the root node to the leaf node. can do.

Creating the trie data structure may include deleting all child nodes of the node with the reference count of '1'.

Generating the clustering model may include automatically performing a predefined action on each cluster in response to identifying clusters of the log sequence using the generated clustering model.

The clustering step may include calculating a difference value between adjacent log sequences according to a result of sorting the log sequences, and clustering the log sequences using whether the difference value exceeds a reference value.

The step of calculating the difference value includes, with respect to the adjacent first log sequence and the second log sequence, identifying the first offset at which a different token appears between the first log sequence and the second log sequence. and determining the offset as a difference value between the first log sequence and the second log sequence.

The reference value may be a value determined according to the alignment result of the log sequences and a user input to a user interface including a reference value change control.

The user interface may display a changed clustering result when a user input for changing the reference value is applied through the reference value change control.

The user interface may display the changed clustering result by overlaying the area of each cluster on the alignment result of the log sequences when a user input for changing the reference value is applied through the reference value change control. When a user input to change the reference value is applied, the changed clustering result may be displayed by overlaying the area of each cluster on the alignment result of the log sequences.

The step of masking some of the tokens of the log sequence according to another embodiment of the present disclosure is to input the masked log sequence into a clustering model in the form of a trie data structure with each token as a node, It may include identifying clusters in the log sequence.

The masking step may include identifying a masking target token for each of the plurality of log sequences, and replacing the masking target token with a masking expression corresponding to the type of the masking target token. there is.

Each node of the trie data structure has a cluster reference number, and the step of identifying a cluster of the log sequence involves node traversing by matching tokens of the masked log sequence with each node of the clustering model. ), and when a node with a reference count value of 1 is reached, the node search may be terminated, and the cluster indicated by the node at the end may be identified as the cluster of the masked log sequence. .

The log data management method may further include automatically performing a predefined action on each cluster in response to identifying a cluster of the log sequence.

A log data management system according to another embodiment of the present disclosure for solving the above technical problem includes a network interface for receiving log data including a plurality of log sequences, a memory into which a clustering modeling program is loaded, and a memory loaded into the memory. It may include one or more processors that execute a clustering modeling program. At this time, the clustering modeling program uses an instruction for masking some of the tokens of each log sequence included in the log data, an instruction for sorting each masked log sequence, and the alignment result of the log sequences. Thus, an instruction for clustering log sequences may be included.

A log data management system according to another embodiment of the present disclosure for solving the above technical problem includes a network interface that receives a log sequence, a memory in which a clustering program is loaded, and one or more executing the clustering program loaded in the memory. It includes a processor, and the clustering program includes instructions for masking some of the tokens of the log sequence and processing the masked log sequence into a clustering model in the form of a trie data structure with each token as a node. An instruction that, by inputting, performs at least one of identifying a cluster of the log sequence, outputting the identified cluster of the log sequence, and attaching information about the identified cluster to the log sequence. may include.

1 illustrates an example environment in which a log data processing and classification system according to some embodiments of the present disclosure may be applied.
Figure 2 is a block diagram for explaining a log data processing device according to an embodiment of the present disclosure.
Figure 3 is a flow chart of a cluster model generation method according to another embodiment of the present disclosure.
FIG. 4 is a diagram illustrating log data sequence boundary designation that can be performed in some embodiments of the present disclosure.
FIG. 5 is a diagram illustrating a method of tokenizing and masking log data that can be performed in some embodiments of the present disclosure.
FIG. 6 is a flowchart for explaining in detail some of the operations described with reference to FIG. 1 .
FIG. 7 is a diagram illustrating a user interface including a screen for displaying alignment results of log sequences and a reference value change control that can be displayed in some embodiments of the present disclosure.
FIG. 8 is a diagram illustrating a method of clustering a plurality of log data based on a designated reference value that can be performed in some embodiments of the present disclosure.
9 is a diagram illustrating a log data sequence clustered through linear comparison, which may be displayed in some embodiments of the present disclosure.
10 is a diagram illustrating segmented log data clusters that may be displayed in some embodiments of the present disclosure.
FIG. 11 is a flowchart for explaining in detail some of the operations described with reference to FIG. 1.
FIG. 12 is a flowchart for explaining in detail some of the operations described with reference to FIG. 11.
13 is an exemplary TRIE data structure that can be generated as a cluster model as a result of performing some embodiments of the present disclosure.
Figure 14 is a flowchart of a new log data classification method according to another embodiment of the present disclosure.
Figure 15 is a diagram illustrating a trie data structure to explain a method of searching for clusters similar to newly inserted log data.
16 is a hardware configuration diagram of a computing system that can be used as a component in some embodiments of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the technical idea of the present invention is not limited to the following embodiments and may be implemented in various different forms. The following examples are merely intended to complete the technical idea of the present invention and to be used in the technical field to which the present invention pertains. It is provided to fully inform those skilled in the art of the scope of the present invention, and the technical idea of the present invention is only defined by the scope of the claims.

In describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, several embodiments of the present disclosure will be described with reference to the drawings.

1 illustrates an example environment in which a log data management system according to some embodiments of the present disclosure may be applied. Figure 1 shows one log data output system 200 connected to the network, but this is only for convenience of understanding, and the number of log data output systems 200 may vary.

Figure 1 only shows a preferred embodiment for achieving the purpose of the present disclosure, and some components may be added or deleted as needed. Additionally, note that the components of the exemplary environment shown in FIG. 1 represent functional elements that are functionally distinct, and that a plurality of components may be implemented in an integrated form in an actual physical environment.

For example, the log data management system 100 and the log data output system 200 may be implemented in the form of different logic within the same computing device, and the log generated by the log data output system 200 The log data management system 100 may process data within the same computing device. Hereinafter, each component shown in FIG. 1 will be described in more detail.

The log data management system 100 may acquire a plurality of log data generated by the log data output system 200 and cluster the acquired plurality of log data. Additionally, the log data management system 100 may create a log data clustering model using the clustering results of a plurality of log data. As shown in FIG. 2, the above-described operations may be performed in the log data clustering model generator 110 included in the log data management system 100.

Additionally, the log data management system 100 may extract attribute values of log data included in target log data using a log data clustering model. As shown in FIG. 2, the above-described operation may be performed in the log data clustering unit 120 included in the log data management system 100.

Log data management system 100 may be implemented with one or more computing devices. For example, all functions of log data management system 100 may be implemented on a single computing device. As another example, the first function of the log data management system 100 may be implemented in a first computing device, and the second function may be implemented in a second computing device. Here, at least one of the first computing device and the second computing device may be a cloud server.

Additionally, the computing device may be a laptop, desktop, laptop, etc., but is not limited thereto and may include all types of devices equipped with a computing function. However, in an environment where the log data processing device 100 must cluster log data by linking with various log data output systems 200, it may be desirable for the log data management system 100 to be implemented as a high-performance server-class computing device. there is. An example of a computing device will be described later with reference to FIG. 16.

Meanwhile, each component 110, 120, and 130 shown in FIG. 2 may mean software or hardware such as FPGA (Field Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit). . However, the components are not limited to software or hardware, and may be configured to reside in an addressable storage medium, and may be configured to execute one or more processors. The functions provided within the above components may be implemented by more detailed components, or may be implemented as a single component that performs a specific function by combining multiple components.

A specific method by which the log data management system 100 clusters log data will be described in detail later with reference to the drawings of FIG. 3 and below.

Next, the log data output system 200 may perform various operations and generate log data. For example, the log data output system 200 may be an IoT (Internet of Things) device or a device included in a manufacturing facility. However, the scope of the present disclosure is not limited to the preceding examples, and the log data output system 200 generates log data by recording data about performed actions and events, but may refer to all things implemented as computing devices. You can.

Next, the user terminal 300 may load the log data sequence clustering model generated by the log data management system 100. Additionally, the user terminal 300 may load attribute values of log data extracted by the log data management system 100. The user terminal 300 may have a web browser or a dedicated application installed in order to display data loaded from the log data management system 100 to the user. For example, the user terminal 300 may be installed on a desktop. (Desktop), Workstation, Laptop. It may be either a tablet or a smart phone, but is not limited to the previous examples and may include all types of devices equipped with computing functions.

In some embodiments, log data management system 100, log data output system 200, and user terminal 300 may communicate over a network. The network is a local area network (LAN), a wide area network (WAN), a mobile radio communication network, a mobile radio communication network such as Wibro (Wireless Boradband Internet), Wibro ( It can be implemented with all types of wired/wireless networks such as Wireless Broadband Internet).

So far, the configuration and operation of the log data processing device 100 according to some embodiments of the present disclosure have been described with reference to FIGS. 1 and 2. Hereinafter, methods according to various embodiments of the present disclosure will be described in detail with reference to FIGS. 3 to 10.

Each step of the methods according to some embodiments of the present disclosure may be performed by a computing device. In other words, each step of the above methods may be implemented as one or more instructions executed by a processor of a computing device. All steps included in the methods may be performed by a single physical computing device, but the first steps of the method are performed by a first computing device and the second steps of the method are performed by a second computing device. It may also be performed by. Hereinafter, the description will be continued assuming that each step of the above methods is performed by the log data processing device 100 illustrated in FIG. 1. However, for convenience of explanation, the description of the operator of each step included in the above methods may be omitted.

In step S100 of FIG. 3, the boundary of the log data sequence may be designated by the log data management system based on a common pattern. Please refer to FIG. 4 to explain the step in more detail. Here, the log data sequence may be log data output while the log data output system performs a specific operation. For example, if the log data output system outputs one log data consisting of two lines while performing the first operation, the log data is one sequence.

If log data is generated from one log data output system, a common pattern may exist for all log data sequences. For example, it can be seen that all log data in FIG. 4 have the same pattern of 'log data generation date and time - classification (INFO/WARN) [Message 1] - Message 2'.

In order to specify that the log data are different log data sequences output by different operations, the boundaries of the log data need to be specified. For example, since the log data in FIG. 4 includes not only one line but also two lines of log data, even if the log data is input in different lines, it can be one log data sequence.

Therefore, like the first log data (5) in FIG. 4, a line break symbol '\n' is inserted at the end of the log data. Accordingly, in the log data of FIG. 4, the boundary of the log data sequence may be designated based on '\n'.

The sequence boundary designation method is not limited to '\n', and other sequence boundary designation delimiters can be specified through environment settings, and adjacent log data sequences are automatically selected based on the delimiter. Logic to distinguish may be performed.

In some cases, the log data sequence may be divided by the user selecting the boundary of the log data sequence. In this case, by providing a standard for distinguishing different log data, the log data sequences to be used for clustering can be accurately distinguished.

Hereinafter, a case where the log data management system performs log data clustering without specifying log data sequence boundaries will be described. When a plurality of different log data are recognized as the same log data sequence, the problem of lowering the accuracy of clustering may occur. At this time, a delay in clustering time may occur.

According to this embodiment, the log data management system determines the exact boundary between log data, specifies that the log data before the boundary and the log data behind the boundary are different log data sequences, and based on this, creates a plurality of log data sequences. By performing clustering, clustering accuracy can be improved and clustering time can be reduced.

Next, in step S200, the log data management system may separate individual log sequences into a plurality of tokens. Tokens can be expressed as spaces, special characters (!, @, #, etc.), numbers, alphabets, etc. To explain individual log sequence tokenization in more detail, it will be described with reference to FIG. 5.

For example, referring to Figure 5, the string '2015-07-20 - INFO [main:Quorum]' is 2015, -, 07, 29, (space), -, (space), INFO, (space), It can be separated into [, main, :, Quorum, ], etc. In this way, the log data management system does not separate each number from a meaningful continuous string that can infer the year of occurrence, such as '2015'. Additionally, special characters such as '-' or '(space)' and spaces can be separated into different tokens.

If the log data management system performs log data sequence clustering without performing tokenization, a delay in clustering time may occur because meaningful strings in the entire log data string must be recognized each time clustering is performed.

According to this embodiment, in separating individual log sequences, the log data management system replaces meaningless strings when they exist as individual tokens such as spaces and special characters with meaningful information such as log data occurrence year, log data occurrence date, and log data classification. By distinguishing the strings it contains, the log data sequence clustering execution time can be reduced.

Next, in step S300, the log data management system may mask individual tokens of the log sequence. To explain in more detail the step of masking individual tokens of a log sequence, the description will be made with reference to FIG. 4 or FIG. 5 . Here, masking means replacing individual tokens of a log sequence with one consistent word even if the strings they contain are different from each other if the types of strings are the same.

For example, referring to Figure 5, the string 2015-07-20 - INFO [main:Quorum]' is 2015, -, 07, 29, (space), -, (space), INFO, (space), [ For results separated by tokens such as , main, :, Quorum, ], the '2015' token, '07' and '20' tokens contain different strings. However, based on the commonality that the token is composed of numbers, it can be equally replaced with '\d+'.

In some embodiments related to step S300, the log data management system may mask values that do not play a major role in the clustering process among individual tokens and may lead to incorrect results.

For example, referring to Figure 4, '2015-07-29 17:41:41,728 - INFO [WorkerReceiver[myid=1]:FastLeaderElection@542] - Notification: 1 (n.leader), 0x0 (n.zxid ), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)' The log data sequence is ' 2015-07-29 17:41:41,932 - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:QuorumCnxManager@368] - Cannot open channel to 2 at election Like the 'address /10.10.34.12:3888' log data sequence, log data belonging to the WARN classification may be clustered into the same log data cluster based on the fact that the log data sequence of the INFO classification occurred in the same year as the log data sequence.

If incorrect clustering is performed as described above, the user may later incur the inconvenience of having to correct the incorrect clustering one by one, resulting in inefficiency of the log data sequence clustering method. According to this embodiment, the log data management system may avoid an error in which logs of different types or classifications are clustered into the same cluster due to the same date, time, or other similar numerical values.

In some other embodiments related to step S300, the log data management system may replace all tokens consisting of only numbers with '\d+'. Here, the original and masked data can be stored separately and processed to refer to each other. For example, referring to Figure 5, the original 2015 string (10) may be replaced with '\d+'.

According to this embodiment, the log data management system may avoid an error in which log data belonging to a specific cluster and one or more log data sequences of different types are clustered together by individual tokens consisting of numbers included in the log data.

In some other embodiments related to step S300, tokens consisting of strings rather than numbers may be replaced with [a-zA-Z]+. Here, the log data management system stores the original and masked data separately and can process them so that they are referenced to each other.

For example, referring to Figure 5, in the '[main:Quorum]' token (11), the 'main' and 'Quorum' tokens can both be replaced with '[a-zA-Z]', as in component 11a. There is also. According to this embodiment, the log data management system can mask individual tokens comprised of strings rather than numbers included in log data. As a result, it is possible to avoid errors in which log data belonging to a specific cluster and one or more log data sequences of different types are clustered together.

In some other embodiments related to step S200, tokens consisting of special characters and spaces may be replaced with the corresponding special characters and '\s', respectively. Here, the log data management system stores the original and masked data separately and can process them to refer to each other.

For example, referring to Figure 5, each of the individual tokens '[', ':', and ']' separated from the '[main:Quorum]' string (11) of the log data source remains even after going through the log data masking step. The original special characters are written as they are without being replaced with separate symbols. In addition, the blank space 12 in the original may be replaced with a symbol '\s' meaning a blank space, as in component 12a, but the scope of the present disclosure is not limited to the example shown in FIG. 5, and log data It should be noted that all symbols whose string format can be distinguished based on the format of the string included in may be included in the scope of the present disclosure.

According to this embodiment, the log data management system can mask individual tokens consisting of special characters or spaces included in log data. As a result, it is possible to avoid errors in which log data belonging to a specific cluster and one or more log data sequences of different types are clustered together.

Next, in step S400, the log data management system may sort log data sequences based on the separately stored original and masking data. Here, the system may use a method used for general string data sorting to sort the log data sequence.

If clustering is performed without performing a sorting operation, the complexity of calculations for clustering log data increases, resulting in clustering taking excessive time. According to this embodiment, a plurality of log data sequences are sorted based on strings, so that when the log data management system performs clustering between similar log data sequences, the complexity of the clustering calculation is reduced, the clustering time is reduced, and the clustering accuracy is improved. It can be improved.

In some embodiments related to step S400, the log data management system may use the masked log data as a reference when performing the log data sequence sorting operation. Here, since log data is usually written from left to right, it can be sorted based on the left character. However, in the case of left-right letters such as Arabic letters, they may be sorted based on the letter on the far right.

Next, in step S500, the user may determine a reference value for log data sequence clustering. In order to explain in more detail the step of the user determining the reference value, it will be described with reference to FIGS. 7 and 8.

Here, the threshold may mean the maximum location information of the token matching the log data pattern. For example, referring to FIG. 8, the log data sequence corresponding to data[2] and the log data sequence corresponding to data[5] are clustered in the same cluster when the user sets the reference value to 58. The log data sequence corresponding to Data[2] is '\d+-\d+-\d+\s\d+:\d+:\d+\S+\d+\s-\s[a It has a masked pattern of -zA-Z]+\S+\[[a-zA-Z]+'. In addition, from the left of data[5] to the 58 standard value token location information, '\d+-\d+-\d+\s\d+:\d+:\d+\S+\d+\s-\s[a-zA-Z]+ Since they have the same masked pattern of \S+\[[a-zA-Z]+', data[2] and data[5] can be clustered into the same cluster.

In some embodiments related to step S400, the log data management system may receive input of a reference value from the user through a user interface that includes a control for changing the reference value and alignment results of log data sequences. For example, referring to FIG. 7, the user can input a threshold value (43) by sliding the threshold value change control (40) using the mouse input (42). According to this embodiment, the user can easily perform log data sequence clustering based on inputting a reference value through an intuitive interface without entering a separate code.

In some other embodiments related to step S400, the log data management system outputs the clustering results of log data sequences based on original log data rather than masked log data in the user interface in real time, allowing the user to immediately intuitively view the results. You can also check it with . According to this embodiment, the user can check the clustering result based on the original log data in real time and immediately adjust the reference value if the clustering result of the log data sequences is not appropriate.

Next, in step S600, the log data management system may perform log data sequence clustering based on a reference value determined by the user. In order to explain the step of performing log data sequence clustering in more detail, it will be described with reference to FIGS. 6 and 8 to 9.

In step S610 of FIG. 6, the log data management system may calculate a difference value between a specific log data sequence and an adjacent log data sequence. Here, the similarity measurement method implemented in the present disclosure may use the location of the point where a specific log data sequence first differs by comparing masked data with masked data adjacent to it. For example, in the strings 'abcd' and 'abxy', the third character (c and x) of the two data is different, so the similarity between the two data is 2. However, the initial value of similarity is 0.

When using the existing string similarity measurement method, the entire data[N] must be searched for similarity analysis excluding the area replaced through data categorizing, so the minimum O( ) cost. According to this embodiment, since the data is sorted based on the regular expression pattern as described above, the similarity between data data[i] masking a specific log data sequence and data[i+1], which is adjacent log sequence masking data, is calculated to obtain O( Since only N) costs are required, it can provide a method of generating a log data cluster classification model with significantly improved time complexity.

In some other embodiments related to step S610, the log data management system moves p until token[p] of the log sequence masking data of data[i] and data[i+1] match, and then moves p until the point where they do not match. p can be recorded in the similarity matrix. Here, token[p] refers to the pth token of log data, and the p variable may be a threshold determined by the user. At this time, the minimum value among the length values of data[i] and data[i+1] can be taken as the maximum value of p.

For example, referring to Figure 8, when the reference value (31) determined by the user is 58, token[58] of data[1] is '+' and token[58] of data[2] is '+'. When the reference value (31) is 58, they can be clustered into one cluster, but when the reference value is 60, they are separated into different clusters, so the similarity between data[1] and data[2] is recorded as 58 in the similarity matrix.

Next, in step S620, the log data management system determines that if the log data sequences data[i-1] and data[i] are clustered into the same cluster, data[i] and data[i+1] are clustered into the same cluster, data It can be determined that [i-1] and data[i+1] can be clustered into the same cluster, so the clusters can be classified through linear comparison.

In some embodiments related to step S620, referring to FIGS. 8 and 9, the log data management system normalizes data[1] and data[2], assuming that the user specifies the reference value as 60, and then normalizes the two data Since the longest length of matching tokens is 58, the two data can be clustered into different clusters. However, in the case of data[2], data[3], data[4], and data[5], the longest lengths of matching tokens with data[2] are 89, 110, and 110, respectively, which is a similarity greater than the above reference value of 60. It can be clustered into one cluster (30) related to data[2]. In Figure 9, it can be seen that data[3] also has the same token as data[2] and token[110]. Likewise, referring to Figures 8 and 9, data[6] to data[9] are also clustered into one cluster according to the same logic.

When the log data management system performs clustering by individually measuring the similarity between data[i] and data[i+1] to data[i+n] (n is the end point of the log data sequence), the corresponding algorithm The time complexity exceeds O(N), making log data clustering very inefficient. According to this embodiment, the log data management system operates when the log data sequences data[i-1] and data[i] are clustered into the same cluster, and data[i] and data[i+1] are clustered into the same cluster, data[ i-1] and data[i+1] can also be determined to be clustered into the same cluster, which can provide significant cost savings in log data sequence clustering execution time.

The algorithm corresponding to the clustering method related to step S600 described so far may be predetermined, and may be changed from one to another depending on the case. In determining the suitability of log data sequence clustering, which will be explained later in relation to S700, it should be noted that if the clustering result is not suitable, clustering may be performed under conditions different from existing ones.

Next, in step S700, the log data management system may evaluate the suitability of the log data sequence clustering results since the purpose of separating logs for use in log data analysis may always be different in various embodiments.

In some embodiments related to step S700, the log data sequence clustering result may be evaluated by the user, thereby determining at which point the clusters should be separated by the user's judgment. According to this embodiment, by providing a standard for the user to distinguish between different log data, log data sequences can be accurately classified to suit the purpose for which the clustering result is to be used.

Next, in step S800, the log data management system may generate a log data sequence clustering model in response to whether the log data sequence clustering result of the user or the log data management system is suitable. Here, the clustering model is created based on the fact that log data output by a specific log data output system has a common prefix pattern. The step of generating the clustering model (S800) will be described in detail later with reference to FIG. 11.

The operations related to generating the log data sequence clustering model so far may be used independently, but the scope of the present disclosure is not limited thereto, and a plurality of operations may be used together.

The description will be made again with reference to FIGS. 11 to 13.

FIG. 11 is a flowchart for explaining in detail the operation of the log data management system described with reference to FIG. 1 to generate a log data clustering model.

In step S810, the log data management system may create a TRIE data structure with each token of the log data as a node. The step of generating the TRIE data structure (S810) will be described in detail later with reference to FIGS. 12 and 13. Here, the data structure is created based on the fact that log data output by a specific log data output system has a common prefix pattern. Additionally, the node may have the reference count for each log data sequence cluster as an attribute value.

Next, in step S820, the log data management system may determine whether a cluster of log sequences is identified. For example, a method of identifying a cluster of a log sequence may be to confirm that one or more clusters have been created by having two or more log data sequences have a similarity greater than a reference value determined by the user.

Next, in step S830, the log data management system may automatically perform a predefined action on each cluster in response to the cluster of the log sequence being identified. Here, the predefined action may be an action predefined by the user performing a separate action or an initial setting of the log data management system.

In some embodiments related to the predefined action of step S830, the predefined action may be sending an alarm message to the user terminal when a log belonging to a specific log data cluster is found. For example, the log data management system may transmit an alarm message to the user terminal in response to identifying that logs whose classification of log data is 'WARN' are clustered. According to this embodiment, the user can quickly recognize that a critical log has been output to the system and take action before a critical error occurs in the system.

In some embodiments related to the predefined action of step S830, the predefined action is to force the log data management system to initialize the log data output system that outputs the log when a log belonging to a specific log data cluster is found. It could be. For example, the log data management system may initialize the log data output system in response to the user identifying that an unwanted connection has occurred and that the related logs have been clustered. According to this embodiment, the log data management system can respond to security-related problems that may occur from time to time without the user taking manual action.

If the log data management system does not automatically perform predefined actions using log data, users will have to check each clustered log data sequence and manually perform the desired action. According to this embodiment, the log data management system automatically performs actions predefined by the user for each log data sequence cluster without any additional actions by the user, thereby performing tasks utilizing log data more economically. It can provide an effect that can be achieved.

Hereinafter, the step of generating a try data structure will be described in detail with reference to FIGS. 12 and 13.

Referring to FIG. 12, in step S810-1, the log data management system may determine a representative log data sequence for each log data sequence cluster. In some embodiments, the log data management system may extract the representative data by determining one of the shortest length data or longest length data when located at the top or bottom of the cluster, but the entire cluster It should be noted that representative data must be extracted in the same manner as the method determined above.

Next, in step S810-2, the log data management system may tokenize the extracted representative data and store it in a trie data structure. The method of tokenizing the representative data may be the same as the method of tokenizing the log data to create the log data sequence clustering model.

In some embodiments related to step S810-2, referring to FIG. 13, node 50a is a node in which representative log data with a token in the form of '\d+' located at the top is stored. Here, the number in parentheses of the 50a node means the number of clusters referenced by the 50a node.

According to this embodiment, in relation to the step of clustering newly inserted log data, which will be described later, when new log data is input, the log data management system performs regular expression conversion and tokenization of the new data through DFS search of the clustering model. New data can also be separated into clusters with the highest similarity.

Next, in step S810-3, the log data management system may merge two or more consecutive nodes having a single child node among the components of the trie data structure generated by tokenizing the representative data.

In some embodiments related to step S810-3, the log data management system may merge all nodes from the root node to the point where there is one child node. Referring to FIG. 13, nodes 50a, 50b, and 50c have only one child node in the data structure. Here, 50d is a child node of 50c, 50c is a child node of 50b, and 50b is a child node of 50a. When the log data management system searches for clusters similar to the data when inserting a new log data sequence, continuously searching for points with one child node may be a meaningless task. According to this embodiment, the system may maximize memory efficiency by deleting unnecessary nodes of the generated tri data structure.

Next, in step S 810-4, the log data management system may delete all child nodes whose referring cluster number is 1. However, in cases such as node 51, where the reference count is 1 but the parent node is not, it is not deleted.

For example, referring to FIG. 13, node 52 is a child node whose reference cluster number is 1 of the '-(1)' node whose reference cluster number is 1. Even though the upper parent node refers to only one cluster, continuously creating new cluster nodes can cause inefficiencies in clustering execution time and memory capacity.

According to this embodiment, the log data management system deletes meaningless nodes that occupy memory, thereby reducing the number of searches as much as the deleted child node when the log data management system performs clustering of a new log data sequence, and securing memory to save time, Capacity cost savings can also be achieved.

According to the log data management method according to some embodiments of the present disclosure described so far with reference to FIGS. 3 to 13, a log data clustering model can be created to search for a cluster matching a newly inserted log data sequence. . More specifically, based on the fact that there is a certain pattern in the log data output by the log data output system, the log data is tokenized, the log data is masked, and log data sequences with a similarity greater than a user-determined standard are clustered. And, in response to the clustering result being suitable, a log data sequence clustering model having a trie data structure can be created.

Hereinafter, with reference to FIGS. 14 and 15, a detailed description will be given of how the log data management system automatically clusters newly inserted log data using the log data sequence clustering model.

In step S900, new log data may be inserted into the log data management system. Here, newly inserted log data may mean log data newly generated by a log data output system. The log data sequence clustering model can be clearly understood by referring to FIGS. 3 to 13.

Next, the new log data may be tokenized in step S1000. Here, the log data may be expressed as spaces, special characters (!, @, #, etc.), numbers, alphabets, etc. The method of tokenizing log data can be clearly understood by referring to FIG. 5.

Next, in step S1100, the log data management system may mask individual tokens of the tokenized log sequence. Here, masking means replacing individual tokens of a log sequence with one consistent word even if the strings they contain are different from each other if the types of strings are the same. The method for masking the tokens can be clearly understood by referring to FIG. 5 above.

Next, in step S1200, the log data management system inputs the masked new log data sequence into the log data sequence clustering model in the form of a tri data structure with each token as a node, thereby forming a cluster similar to the new log data. You can search.

In some embodiments related to step S1200, the log data management system may end the search when the last token of the new log data is reached while searching for a cluster of a similar type as the new log data.

For example, referring to FIG. 15, assuming that the new log data is input to the clustering model, the log data management system confirms that token[1] of the new log data is the same symbol as T1 (60), and Search the next node, check that token[2] is the same symbol as T2(61), search the next node, check that token[3] is the same symbol as T9(62b), and check that token[3] is the same symbol as T9(62b). By confirming that it is the last token of the data and ending the search, it can be evaluated that the new log data and T9 (62b) are similar clusters.

In some other embodiments related to step S1200, the log data management system may terminate the search when the reference count among the nodes of the data structure is 1 while performing the search.

For example, referring to FIG. 15, assuming that the new log data is input to the clustering model, the log data management system confirms that token[1] of the new log data is the same symbol as T1 (60), and Search the next node, check that token[2] is the same symbol as T2(61), search the next node, check that token[3] is the same symbol as T9(62b), and check the node T9(62b). In response to the reference count being 1, the search may be terminated and the new log data may be evaluated for similarity to T9 62b.

In some other embodiments related to step S1200, the log data management system may not advance to the last token of the new log and terminate the search if there are no more matching tokens.

For example, referring to FIG. 15, assuming that the new log data is input to the clustering model, the log data management system confirms that token[1] of the new log data is the same symbol as T1 (60), and Search the next node, and find that token[2] is the same symbol as T2(61). Search the next node, and find that token[3] is not the same symbol as any of T3(62a) and T9(62b). You can check and end the search and evaluate that no clusters similar to the new log data exist.

Next, in step S1300, the log data management system may evaluate whether a cluster similar to the new log data as a result of the search exists in the clustering model. The evaluation method can be clearly understood by referring to several embodiments related to step S1200.

When performing a tree-type data structure search, if the search is performed without specifying a separate termination rule in the log data management system, the system repeatedly performs a meaningless search. This may cause inefficiency in the time complexity of the log data management method.

According to this embodiment, the log data management system can complete cluster search early without having to inspect all tokens of new log data, thereby significantly improving the time efficiency of performing new log data clustering.

Next, in step S1400, the log data management system cannot proceed on the tree because it no longer searches for a node matching token[p] of the new log data within the log data clustering model, or the node corresponding to token[p] In response to the fact that the nodes were all present but no node with a reference count of 1 was encountered, a separate indication may be added to the new data.

In some embodiments related to step S1400, the unprocessed data may be a case in which the corresponding cluster is not included when creating a model, or may be a case in which the corresponding data is not intentionally included in the clustering model because there is no need to process the data. The scope of the present disclosure is not limited to these examples, and all cases of intentional or unintentional failure to process may be included in the scope of the present disclosure.

According to this embodiment, even when the log data management system fails to classify a cluster of the new log data in the clustering model, a situation in which the system stops operating due to a program error can be prevented.

Next, in step S1500, the log data management system may return the corresponding cluster in response to the existence of a cluster similar to the new log data. Here, a method of returning the cluster may be to provide a separate display in the user interface so that existing clusters and new log data can be visually recognized as the same classification.

It should be noted that the scope of the present disclosure is not limited to this embodiment, and all additional work premised on a response that a cluster similar to new log data exists may be included in the scope of the present disclosure.

16 is a hardware configuration diagram of a computing device according to some embodiments of the present disclosure. For example, the computing device 1000 shown in FIG. 16 may refer to the log data management system 100 described with reference to FIG. 1 . The computing device 1000 includes one or more processors 1100, a system bus 1600, a communication interface 1200, a memory 1400 that loads a computer program 1500 executed by the processor 1100, and It may include a storage 1300 that stores a computer program 1500.

The processor 1100 controls the overall operation of each component of the simulation device 200. The processor 1100 may perform operations on at least one application or program to execute methods/operations according to various embodiments of the present disclosure. The memory 1400 stores various data, commands and/or information. The memory 1400 may load one or more computer programs 1500 from the storage 1300 to execute methods/operations according to various embodiments of the present disclosure. The bus 1600 provides communication between components of the simulation device 200. The communication interface 1200 supports Internet communication of the simulation device 200. Storage 1300 may non-temporarily store one or more computer programs 1500. The computer program 1500 may include one or more instructions implementing methods/operations according to various embodiments of the present disclosure. When the computer program 1500 is loaded into the memory 1400, the processor 1100 can perform methods/operations according to various embodiments of the present disclosure by executing the one or more instructions.

The computer program 1500 includes the following operations: receiving log data including a plurality of log sequences, loading a clustering modeling program, executing the clustering modeling program loaded in the memory, and generating a token of the log sequence. an operation of masking some of the log sequences, an operation of the clustering program inputting the masked log sequence into a clustering model in the form of a tri data structure with each token as a node, thereby identifying a cluster of the log sequence, the log It may include instructions for performing an operation of outputting the identified cluster of a sequence and an operation of attaching information about the identified cluster to the log sequence.

So far, various embodiments of the present disclosure and effects according to the embodiments have been mentioned with reference to FIGS. 1 to 16 . The effects according to the technical idea of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below.

The technical ideas of the present disclosure described so far can be implemented as computer-readable code on a computer-readable medium. The computer program recorded on the computer-readable recording medium can be transmitted to another computing device through a network such as the Internet and installed on the other computing device, and thus can be used on the other computing device.

Although operations are shown in the drawings in a specific order, it should not be understood that the operations must be performed in the specific order shown or sequential order or that all illustrated operations must be performed to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Although embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention can be implemented in other specific forms without changing the technical idea or essential features. I can understand that there is. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the technical ideas defined by this disclosure.

Claims (20)

In a method performed by a computing system,
masking some of the tokens of log data including a plurality of log sequences;
Sorting the masked log sequences; and
Comprising the step of clustering log sequences using the alignment results of the log sequences,
How to manage log data. According to claim 1,
The masking step is,
Identifying a masking target token for each of the plurality of log sequences; and
Comprising replacing the masked token with a masking expression corresponding to the masked token,
How to manage log data. According to clause 2,
The substitution step is,
determining a type of the token to be masked; and
Comprising the step of replacing the masked token with a masking expression corresponding to the type of the masked token,
How to manage log data. According to claim 1,
Further comprising generating a clustering model using the clustering result,
How to manage log data. According to clause 4,
The step of generating the clustering model is,
Including the step of creating a trie data structure with each token as a node,
Each node has the reference count for each cluster as an attribute value,
How to manage log data. According to clause 5,
The step of creating the try data structure is,
determining a representative log sequence for each cluster; and
Including generating the trie data structure using the representative log sequences,
How to manage log data. According to clause 5,
The step of creating the try data structure is,
When there are two or more consecutive nodes having a single child node in the direction from the root node to the leaf node, including merging two or more consecutive nodes having a single child node,
How to manage log data. According to clause 5,
The step of creating the try data structure is,
Including deleting all child nodes of the node having '1' as the reference count,
How to manage log data. According to clause 4,
The step of generating the clustering model is,
In response to identifying clusters of the log sequence using the generated clustering model, automatically performing a predefined action on each cluster,
How to manage log data. According to claim 1,
The clustering step is,
calculating a difference value between adjacent log sequences according to a result of sorting the log sequences; and
Including clustering log sequences using whether the difference value exceeds a reference value,
How to manage log data. According to claim 10,
The step of calculating the difference value is,
With respect to adjacent first log sequences and second log sequences, identifying an initial offset at which a different token appears between the first log sequence and the second log sequence; and
Comprising determining the offset as a difference value between the first log sequence and the second log sequence,
How to manage log data. According to claim 10,
The above standard values are:
A value determined according to the user input to the user interface including the alignment result of the log sequences and the reference value change control,
How to manage log data. According to claim 12,
The user interface is,
When a user input for changing the reference value is applied through the reference value change control, the changed clustering result is displayed,
How to manage log data. According to claim 12,
The user interface is,
When a reference value change user input is applied through the reference value change control, the changed clustering result is displayed by overlaying the area of each cluster on the alignment result of the log sequences,
How to manage log data. In a method performed by a computing system,
masking some of the tokens of the log sequence;
Including the step of identifying a cluster of the log sequence by inputting the masked log sequence into a clustering model in the form of a trie data structure with each token as a node,
How to manage log data. According to claim 15,
The masking step is,
Identifying a masking target token for each of the plurality of log sequences; and
Comprising replacing the masked token with a masking expression corresponding to the type of the masked token,
How to manage log data. According to claim 15,
Each node of the trie data structure has a cluster reference count,
The step of identifying clusters of the log sequence is,
Node traversing is performed by matching the token of the masked log sequence with each node of the clustering model. When a node with a reference count value of 1 is reached, the node traversing is terminated, and the process is terminated. Including identifying the cluster indicated by the node at the time as the cluster of the masked log sequence,
How to manage log data. According to claim 15,
In response to identifying a cluster in the log sequence, automatically performing a predefined action on each cluster,
How to manage log data. In computing systems,
a network interface for receiving log data including a plurality of log sequences;
Memory into which the clustering modeling program is loaded; and
Including one or more processors executing a clustering modeling program loaded in the memory,
The clustering modeling program is,
An instruction for masking some of the tokens of each log sequence included in the log data;
an instruction for sorting each of the masked log sequences; and
Including instructions for clustering log sequences using the alignment results of the log sequences,
Log data management system. In computing systems,
A network interface that receives log sequences;
Memory into which the clustering program is loaded; and
Including one or more processors executing a clustering program loaded in the memory,
The clustering program is,
Instructions for masking some of the tokens of the log sequence;
An instruction for identifying a cluster of the log sequence by inputting the masked log sequence into a clustering model in the form of a trie data structure with each token as a node; and
Comprising instructions that perform at least one of outputting the identified cluster of the log sequence and attaching information about the identified cluster to the log sequence,
Log data management system.

Images