KR20190041192A - Method and device for collecting log based on rule - Google Patents

Abstract

The present invention relates to a rule-based dynamic log collection method capable of improving quality of collected log data while controlling the amount of dynamically collected log in accordance with quality of log generated for each service or application; and to an apparatus thereof. According to the present invention, the method and apparatus set one or more log collection rules including a condition defined based on log feature information and one or more behaviors related to log collection executed when the condition is matched. When the log data generated for each service or application is received, the method and apparatus compare the log feature information of the received log data with each condition defined in the one or more set log collection rules to check a matched log collection rule, and process the received log data in accordance with one or more behaviors defined in the corresponding log collection rule when the matched log collection rule exists.

Description

[0001] The present invention relates to a rule-based dynamic log collection method and apparatus,

The present invention relates to a rule-based dynamic log collection method and apparatus capable of increasing the quality of collected log data while controlling the amount of logs dynamically collected according to the quality of log generated for each service or application.

Logging is a record for tracking problems that occur during the development or operation of a program or for monitoring operational status. Logging is essential for fault management and efficient operation in large-scale distributed systems such as cloud systems.

However, as the number of services or applications running in the cloud system increases, the amount of log generated increases significantly, so if the proper control is not achieved, the amount of log becomes too large and the log storage becomes insufficient, The service quality is adversely affected.

For example, if a service is configured as a small unit of container-based application, processing by external request is handled by interworking several or dozens of applications. In such a case, it is necessary to simultaneously analyze the logs of various applications to trace the operation status of the service or to analyze the failure, and it is possible to draw meaningful conclusions.

1, a cloud system is installed in a plurality of nodes 10 such as a distributed server, and hundreds or thousands of applications 110 are continuously executed. In a distributed environment in which applications operate in conjunction with each other The log data generated in the plurality of applications 110 is recorded in a specific central database (hereinafter referred to as log database 20) in order to extract meaningful conclusions through log analysis.

Accordingly, a significant amount of internal communication and storage resources of the cloud system are consumed to transmit and store logs generated by hundreds or thousands of applications 110 in the plurality of nodes 10 to the central log database 20 .

In particular, in recent cloud environments, services are being developed in the form of MSA (Micro-Service Architecture) rather than the traditional 3-tier architecture. This structure is further enhanced with the assumption that the same functionality is provided, as the number of instances for an MSA application averages more than the number of instances for a 3-tier application. Thus, as the amount of log increases, the control of the amount of log collected is becoming important.

Until now, two approaches have been taken so far to control the log volume.

First, all logs are collected during the development phase, but the log function is restricted in the operational phase so that the application generates less log information in a real operating environment. That is, as shown in FIG. 2, each application 110 generates a log, stores it in the local database 120 provided in the node 10, and periodically stores the log data stored in the local database 120 In the transfer to the central log database 20, the application 110 controls the application source such that certain parts generate logs and certain parts do not generate logs. Since this is a method of controlling the log itself generated by the application 110, it is possible to solve the problem that the log affects the service quality or the amount of log is excessively large. On the other hand, the first method requires a process of selecting which log is collected by each application 110 and which log should not be collected, and since this optional log generation function must be reflected in the service logic, Depending on the situation, it is difficult to dynamically change the amount of log.

The second method is a log sampling method. As shown in FIG. 3, the application 110 generates all logs, and further includes a log collector 130 so that the log collector 130 can collect logs selectively And stores it in the local database 120. This can control the amount of log regardless of the service logic, which is more efficient than the method of controlling by the application 110. [ However, this method also has a problem of manually adjusting the sampling rate for each service, and since a log is selected by a simple statistical method, an important log may be missed. For example, if you set to send only 1 log every 10, if the important log occurs on the 15th, it is likely to be discarded. In particular, since the sampling rate is manually adjusted for each service, Difficult to change dynamically.

Korean Registered Patent No. 10-1671314, registered on October 26, 2016 (Name: Travel Product Search Log Data Analysis System)

An object of the present invention is to provide a rule-based dynamic log collection method and apparatus capable of increasing the quality of collected log data while controlling the log amount dynamically collected according to the quality of log generated for each service or application.

Setting at least one log collection rule consisting of at least one action related to a log collection executed when the condition is matched with a condition defined based on the log characteristic information as a solution to the above-mentioned problem of the present invention; Receiving log data generated for each service or application; Comparing log feature information of the received log data with each condition defined in the set one or more log collection rules and checking matching log collection rules; And processing the received log data according to one or more actions defined in the log collection rule, if there is a matching log collection rule.

In the rule-based dynamic log collection method, the log characteristic information may include at least one of a log level, a log creator, a log generation time, and a log generation position.

In addition, in the rule-based dynamic log collection method, the processing of the log data may include storing the received log data directly regardless of a sampling rate; Storing or deleting the received log data according to a currently set sampling rate; And increasing or decreasing the currently set sampling rate.

In the rule-based dynamic log collection method, the condition of the log collection rule may include at least one of whether log data having specific log characteristic information is generated and the number of times the log data is generated.

The rule-based dynamic log collection method may further include storing or deleting the received log data according to a currently set sampling rate if there is no matching log collection rule.

The rule-based dynamic log collection method may further include setting reference information including at least one of a default value, an upper limit value, and a lower limit value for a sampling rate indicating a frequency of collecting log data before the receiving step And processing the log data within a range defined by the reference information in processing the log data.

In addition, the present invention stores, as means for solving the above-mentioned problems, at least one log collection rule consisting of at least one action related to a condition defined on the basis of log feature information and a log collection performed when the condition is matched A rule analyzer that compares log feature information of log data generated for each service or application with at least one log collection rule set to determine a log collection rule matching the currently received log data; A sampling regulator for adjusting a predetermined sampling rate according to a log collection rule determined to match in the rule analyzer; And a log processor for storing or deleting log data currently received according to a log collection rule determined to match in the rule analyzer or a currently set sampling rate.

The rule-based dynamic log collection device may further include a log analyzer that receives log data generated for each service or application and formats the log data into a predetermined format.

The present invention relates to a log analysis system for analyzing a log generated during execution of a service or an application, and more particularly to a log analysis system for collecting log data generated by a corresponding service or application, A log collection rule including at least one log collection rule including a condition based on the log characteristic information and an action capable of controlling log collection performed when the condition is matched is set for each service or application, When the conditions of the log collection rule are matched, log data is collected according to the sampling rate by performing an action set corresponding to the condition, and the sampling rate is changed according to the log generation situation Number Which can change the amount of the log data dynamically.

Accordingly, the present invention can flexibly adjust the amount of log to meet the log generation situation, and in particular, it is possible to collect the log data with high interest by the operator at a higher frequency, and reduce the collection frequency with respect to the non- It is possible to increase the quality of the collected log data at the same time as the control.

In addition, the present invention can be applied to a process flow of a log agent already installed in a node, without applying a large change to an existing log system.

1 is a block diagram showing a basic structure of a log collection system.
2 is a diagram for explaining a conventional log collection method.
3 is a view for explaining another conventional log collection method.
4 is a block diagram illustrating a rule-based dynamic log collector according to the present invention.
FIG. 5 is a flowchart illustrating a rule-based dynamic log collection method according to the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and the accompanying drawings, detailed description of well-known functions or constructions that may obscure the subject matter of the present invention will be omitted. It should be noted that the same constituent elements are denoted by the same reference numerals as possible throughout the drawings.

The terms and words used in the present specification and claims should not be construed to be limited to ordinary or dictionary meanings and the inventor is not limited to the concept of terminology for describing his or her invention in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention and are not intended to represent all of the technical ideas of the present invention. Therefore, various equivalents It should be understood that water and variations may be present.

Also, terms including ordinal numbers such as first, second, etc. are used to describe various elements, and are used only for the purpose of distinguishing one element from another, Not used. For example, without departing from the scope of the present invention, the second component may be referred to as a first component, and similarly, the first component may also be referred to as a second component.

In addition, when referring to an element as being "connected" or "connected" to another element, it means that it can be connected or connected logically or physically. In other words, it is to be understood that although an element may be directly connected or connected to another element, there may be other elements in between, or indirectly connected or connected.

Also, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. It is also to be understood that the terms such as " comprising " or " having ", as used herein, are intended to specify the presence of stated features, integers, It should be understood that the foregoing does not preclude the presence or addition of other features, numbers, steps, operations, elements, parts, or combinations thereof.

A rule-based dynamic log collection apparatus and method according to the present invention is applied to each node (10) executing one or more applications or services in a distributed based service system such as the one shown in FIG. 1, To collect an appropriate amount of log data. ≪ RTI ID = 0.0 > [0031] < / RTI > In particular, the present invention is applicable to the log sampling method as shown in FIG. 3, and after the application 110 generates all the logs, the logs generated by the application 110 are collected and stored in the local DB 120 .

The configuration and operation of the rule-based dynamic log collection apparatus and method according to the present invention will be described with reference to FIGS. 3 and 4. FIG.

3 is a block diagram illustrating a configuration of a rule-based dynamic log collection apparatus according to an embodiment of the present invention.

3, the rule-based dynamic log collection apparatus 400 according to the present invention sets a log collection rule 410 for each application or service operated in the corresponding node 10, , And stores the logs in the local DB 360 installed in the corresponding node 10, and dynamically adjusts the amount of logs to be stored.

For reference, the log data generated by the application 110 may be divided into a plurality of levels according to their characteristics. For example, an application that is implemented in Java has six levels of log levels: FATAL, ERROR, WARN, INFO, DEBUG, and TRACE. Here, FATAL indicates a log relating to a fatal error that may cause the program to be stopped, ERROR indicates a log of an error not so much that the program is stopped, WARN indicates log data of a potentially dangerous situation, DEBUG represents log data with information necessary for program debugging, and TRACE represents log data with more granular information than DEBUG.

The rule-based dynamic log collection device 400 according to the present invention may define one or more log collection rules 410 for each application or service in consideration of the log generation status in consideration of the log level, And log collection rules (410) according to the current log generation situation are applied to dynamically adjust the log collection amount. For example, ERROR and WARN level log data required by a developer or an operator is collected without omission, while INFO level log data generated during normal operation is collected as little as possible. .

To this end, the log level of the present invention may be divided into six layers as described above, and can be divided into three layers of INFO, WARNING and ERROR more simply.

4, the rule-based dynamic log collection apparatus 400 according to the present invention includes a log analyzer 420, a rule analyzer 430, a sampling regulator 440, And a log processor 450, as shown in FIG.

The log data generated by the application 110 of the corresponding node 10 are all received by the log analyzer 420.

The log analyzer 420 receives and formats log data generated from one or more applications 110. Specifically, the normal log data includes at least one of a log creator (service or application name), a log generation date, a log level (for example, INFO, WARN, ERROR) , There is no fixed form or format, so the log data format may be different for each application or service. Thus, the log analyzer 420 receives log data generated from one or more applications 110 and formats it into a predetermined format.

Log data formatted in the log analyzer 420 is input to the rule analyzer 430. [ The rule interpreter 430 compares the input log data with a preset log collection rule 410 to check log collection rules matching the current log generation status.

The log collection rule 410 is defined as {condition, action} so that the amount of log collection can be adjusted dynamically according to the log generation situation. The " condition " may include one or more log characteristic information (e.g., a log creator, a log level, a log generation location, etc.) as a condition to which the rule is applied. The above-mentioned " action " is an action related to log collection, specifically, a determination of the frequency of storing or collecting logs.

That is, the rule analyzer 430 compares the log feature information of the formatted log data with the log collection rules defined as described above, and checks the log collection rules matching the currently received log data.

The rule interpreter 430 transmits a log collection rule matching the log data currently generated to the sampling regulator 440 and the log processor 450. The log analyzer 440 and the log processor 450 compare the log Performs the actions defined in the collection rules.

Specifically, the log processor 450 processes the received log data according to the action set in the matching log collection rule identified by the rule analyzer 430. [ Here, the processing of the log data includes an operation of directly storing the log data in the local DB 460, discarding it immediately without storing it, or selectively storing the log data according to the set sampling rate.

The sampling coordinator 440 changes the sampling rate set in the log processor 450 in accordance with the action set in the matching log collection rule confirmed by the rule analyzer 430. [

In other words, the rule-based dynamic log collecting apparatus 400 according to the present invention can directly store log data generated by the application 110 according to a log collection rule, discard log data without storing it immediately, or sort the log data according to a set sampling rate As a result, it is possible to dynamically adjust the log collection amount according to the log generation situation.

Hereinafter, a dynamic log collection operation according to the rule-based dynamic log collection apparatus 400 according to the present invention will be described by way of example.

In the present invention, log collection rules 410 defined as conditions and behaviors may be more specifically set based on the log level of log feature information, in which case " if N logs of log level A persist , The log is saved / deleted, and the sampling rate is increased or decreased by P%.

The following is an example of a log collection rule set in the above-described format for a specific application. For reference, one or more log collection rules may be set for each application or service.

Rule1 = {INFO≥1000, 0, -1}: Reduces the sampling rate by 1% when 1000 logs of INFO level occur consecutively.

Rule2 = {WARNING, 1, +1}: WARNING If a log of level is encountered once, collect the log and increase the sampling rate by 1%.

Rule3 = {ERROR, 1, +5}: Collect ERROR level log once, and increase the sampling rate by 5%.

In the above example, the log collection rule consists of {condition, storage status, sampling adjustment value}. Here, the storage state and the sampling adjustment value are action information for determining the storage and sampling rate. Particularly, if the storage state is 0, it means that it is stored according to the set sampling rate, and when the storage state is 1, it means that it is stored regardless of the sampling rate.

If the log data of the ERROR level is received from the specific application 110, the rule analyzer 430 extracts the rule 3 with matching log collection rules and sends the rule 3 to the sampling regulator 440 and the log processor 450 And the sampling coordinator 440 increases the sampling rate set in the log processor 450 to 5%, and the log processor 450 stores the log data of the received ERROR level in the local DB 460.

Conversely, when 1000 consecutive INFO level logs are generated from the specific application 110, the rule interpreter 430 extracts the rule 1 and transfers it to the sampling coordinator 440, and transmits the log data to the log processor 450 . Accordingly, the sampling regulator 440 reduces the sampling rate set in the log processor 450 by 1%, and the log processor 450 selects the log data at the INFO level received at the sampling rate reduced by 1% (460).

According to the above rule, when only the INFO level log data is continuously generated in the application 110, the sampling rate is gradually decreased, and the log data amount collected in the log collecting apparatus 400 according to the present invention is gradually And if log data of a WARNING or ERROR level is generated, the amount of log data collected in the log collecting apparatus 400 increases as the sampling rate increases.

Generally, log data generated when an application or a service is normally operated is merely for reporting, and is unnecessary data from the viewpoint of an operator or a developer. Therefore, it is only necessary to confirm that it is in a normal state. On the other hand, in the event of a failure, even log data at the INFO level is likely to contain log information showing that the service quality is not better than normal. Therefore, it is desirable that the log data at the WARING or ERROR level be collected and at the same time, the frequency of collecting the log data at the INFO level should be further increased when the log of the WARING or ERROR level is generated. , Meaningful log data can be collected.

In addition, the rule-based dynamic log collection device 400 according to the present invention collects log data based on a log collection rule by using at least one of a default value, an upper limit value and a lower limit value for the sampling rate So that the dynamic variation of the sampling rate can be made within a certain range.

Next, FIG. 5 is a flowchart illustrating a rule-based dynamic log collection method according to the present invention.

A rule-based dynamic log collection method, which is executed through the log collection device 400 according to the present invention, will be described in the order of operations.

First, according to the present invention, one or more log collection rules 410 are set for each application or service in order to collect rules-based dynamic logs (S105). The log collection rule 410 is defined as a condition that defines a criterion for applying a rule based on one or more pieces of log characteristic information and an action related to a log collection applied when the condition is matched. The action refers to an operation of determining the storage and sampling rate of log data.

In addition, at least one of a default value, an upper limit value, and a lower limit value is set as reference information of a sampling rate that is dynamically changed according to the present invention (S110).

In this state, when one or more applications 110 are executed in the node 10 and log data is generated, log data generated in the one or more applications 110 is received in real time via the log analyzer 420 (S115). At this time, the log analyzer 420 may format the received log data into a predetermined format so that comparison of the log feature information is easy.

The rule-based dynamic log collecting apparatus 400 according to the present invention analyzes the log data received through the rule analyzer 430 when the log data is received, Is present (S120). This can be done by comparing the log feature information included in the log data with the respective conditions defined in the one or more log collection rules 410. In addition, for the comparison, the rule analyzer 430 may further perform statistical processing such as counting based on log feature information.

As a result of the checking in step S120, if there is a log collection rule to be applied, the sampling regulator 440 and / or the log processor 450 changes the set sampling rate or stores or deletes the received log data S125).

On the other hand, if it is determined in step S120 that there is no log collection rule to be applied, the log processor 450 stores or deletes the received log data according to the currently set sampling rate (S130).

Steps S115 to S130 described above are repeatedly executed each time log data is received until the log collection function is terminated (S135).

Based on the above, the rule-based dynamic log collection apparatus 400 according to the present invention can collect log data while dynamically changing the sampling rate according to the log generation status.

The above-described rule-based dynamic log collection method according to the present invention can be implemented in the form of software readable by various computer means and recorded in a computer-readable recording medium. Here, the recording medium may include program commands, data files, data structures, and the like, alone or in combination. Program instructions to be recorded on a recording medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. For example, the recording medium may be an optical recording medium such as a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, a compact disk read only memory (CD-ROM), a digital video disk (DVD) Includes a hardware device that is specially configured to store and execute program instructions such as a magneto-optical medium such as a floppy disk and a ROM, a random access memory (RAM), a flash memory, do. Examples of program instructions may include machine language code such as those generated by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like. Such a hardware device may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to be exemplary and explanatory only and is not restrictive of the invention, It is self-evident to those who have knowledge. Furthermore, although specific terms are used in this specification and the drawings, they are used in a generic sense only to facilitate the description of the invention and to facilitate understanding of the invention, and are not intended to limit the scope of the invention.

Likewise, although the operations are depicted in the drawings in a particular order, it should be understood that such operations must be performed in that particular order or sequential order shown to achieve the desired result, or that all illustrated operations should be performed. In certain cases, multitasking and parallel processing may be advantageous. Also, the separation of the various system components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems are generally integrated together into a single software product, It should be understood.

The rule-based dynamic log collection apparatus and method according to the present invention can be applied to a log analysis system for analyzing a log generated during execution of a service or an application. In particular, So that log data generated by the corresponding service or application can be collected.

In particular, the present invention relates to a method and apparatus for collecting log data on a service or application basis, in which a log collection rule consisting of a condition based on log feature information and an action capable of controlling log collection performed when the condition is matched, And monitors the log data generated by the corresponding applications or the respective applications in the service to perform an action set corresponding to the condition when the conditions of the log collection rule are matched, While collecting log data, it is possible to dynamically change the amount of log data collected by changing the sampling rate according to the log generation situation.

Accordingly, the present invention can flexibly adjust the amount of log to meet the log generation situation, and in particular, it is possible to collect the log data with higher interest by the operator at a higher frequency, and reduce the collection frequency for the non- It is possible to increase the quality of the collected log data at the same time as the control.

In addition, the present invention can be applied to a process flow of a log agent already installed in a node, without applying a large change to an existing log system.

400: rule-based dynamic log collector
410: Log Collection Rules
420: log analyzer
430: Rule interpreter
440: Sampling regulator
450: log processor

Claims (12)

Setting at least one log collection rule consisting of a condition defined based on log characteristic information and at least one action related to log collection performed when the condition is matched;
Receiving log data generated for each service or application;
Comparing log feature information of the received log data with each condition defined in the set one or more log collection rules and checking matching log collection rules; And
And processing the received log data according to one or more actions defined in the log collection rule, if there is a matching log collection rule. The method according to claim 1,
Wherein the log feature information includes at least one of a log level, a log creator, a log generation time, and a log generation position. The method of claim 1, wherein the processing of the log data comprises:
Immediately storing the received log data regardless of a sampling rate;
Storing or deleting the received log data according to a currently set sampling rate; And
And increasing or decreasing the currently set sampling rate. The method according to claim 1,
Wherein the condition of the log collection rule includes at least one of whether or not log data having specific log characteristic information is generated and the number of times the log data is generated. The method according to claim 1,
Further comprising the step of storing or deleting the received log data according to a currently set sampling rate if there is no matching log collection rule. The method according to claim 1,
Further comprising setting reference information including at least one of a default value, an upper limit value, and a lower limit value for a sampling rate indicating a frequency of collecting log data before the receiving step,
Wherein the step of processing the log data comprises processing the log data within a range defined by the reference information. One or more log collection rules consisting of a condition defined on the basis of the log characteristic information and one or more actions related to log collection performed when the conditions are matched are stored and the log characteristic information A rule analyzer for comparing log collection rules with at least one log collection rule that is set in advance, and checking a log collection rule matching the currently received log data;
A sampling regulator for adjusting a predetermined sampling rate according to a log collection rule determined to match in the rule analyzer; And
And a log processor for storing or deleting currently received log data according to a log collection rule determined to match in the rule analyzer or a currently set sampling rate. 8. The method of claim 7,
Further comprising a log analyzer for receiving log data generated for each service or application and formatting the log data into a predetermined format. 8. The method of claim 7,
Wherein the log feature information includes at least one of a log level, a log creator, a log generation time, and a log generation position. 8. The method of claim 7,
The act of the log collection rule may include storing the received log data immediately regardless of the sampling rate, storing or deleting the received log data according to the currently set sampling rate, or changing the currently set sampling rate Based dynamic log collection device. 8. The method of claim 7,
Wherein the condition of the log collection rule includes at least one of whether or not log data having specific log feature information is generated and the number of times the log data is generated. 8. The apparatus of claim 7, wherein the log processor
And if the matching log collection rule does not exist, the received log data is stored or deleted according to the currently set sampling rate.

Images