KR102189127B1 - A unit and method for processing rule based action - Google Patents

Abstract

In the present invention, logs generated in a large amount are defined as actions and processed in a standardized form, and the continuity of actions is defined as a rule to detect abnormal access, suitable for artificial intelligence learning, and capable of processing at high speed. It is an object of the present invention to provide a behavior-based rule processing apparatus and a processing method thereof.
In order to achieve the above object, the behavior-based rule processing apparatus according to the present invention includes a log collection unit that collects logs used for rule processing from a plurality of agents and stores the collected logs, and the contents of the logs. In addition to including a rule block that classifies according to a combination of predetermined values and a condition block that classifies predetermined metadata values of the log, the rule block and the condition check block A real-time rule processing unit that pre-defines the actions of the log as a combination as a rule, and calls the log stored in the log collection unit by real-time distributed streaming.

Description

Behavior-based rule processing device and its processing method {A UNIT AND METHOD FOR PROCESSING RULE BASED ACTION}

The present invention relates to a behavior-based rule processing apparatus and a processing method thereof, and more particularly, to a behavior-based rule processing apparatus and a processing method suitable for artificial intelligence learning and real-time analysis.

In recent years, due to the rapid growth of hardware and the expansion of services provided through the web, the amount of data used in the Internet network is increasing rapidly. Accordingly, abnormal access for the purpose of attacking as well as the access of users who normally use the service are exploding.

Existing analysis systems used to detect abnormal access have determined threats by extracting abnormal information found in specific actions, that is, logs, based on rules designated by the administrator. However, in the traditional log analysis system, it is not able to handle the explosively increasing amount of data in the log, or it has limitations in being able to respond to attack patterns that are becoming more intelligent.

As a method of overcoming such large-capacity processing limitations, most analysis systems adopt a method to increase hardware performance, but limitations such as bandwidth limitations and IO (I/O) delays still exist, and are difficult to judge due to intelligence. In order to detect patterns, researches such as extracting patterns through artificial intelligence models are being conducted, but most of the rule definition methods used in rule engines that process existing rules are programming methods, which can be used in artificial intelligence models. There was a problem that it could not be formulated into a form.

For example, in a general analysis step (rule engine), the normalization log is first stored in a storage medium such as a database, and the stored information is fetched to perform analysis logic (rule processing). However, even if a real-time streaming system is used, this method is not suitable for real-time processing because it requires a process of storing streaming results in a database before analysis (rule processing), and processing time for database queries in the analysis stage.

In addition, in case of various log types, log normalization result items for each log type are designated as columns of each table and stored in the database table. In the analysis logic, the processing of JOIN queries for multiple tables takes a long time. It is not suitable for real-time analysis because it is necessary.

In addition, rules in the rule processing engine must be defined in the form of a query (SQL Query), and are not suitable for an AI learning model for rule generation and rule advancement. Because the result of AI learning is a rule in the form of SQL queries, which is a kind of programming language, it is necessary to implement a learning model close to natural language learning.

Accordingly, the present invention has been proposed to meet the above demands, and its purpose is to define a large amount of logs as actions and process them in a standardized form, and to detect abnormal access by defining the continuity of actions as a rule, It is to provide an action-based rule processing apparatus and a processing method suitable for artificial intelligence learning and capable of processing at high speed.

In order to achieve the above object, the behavior-based rule processing apparatus according to the present invention includes a log collection unit that collects logs used for rule processing from a plurality of agents and stores the collected logs, and the contents of the logs. In addition to including a rule block that classifies according to a combination of predetermined values and a condition block that classifies predetermined metadata values of the log, the rule block and the condition check block A real-time rule processing unit that pre-defines the actions of the log as a combination as a rule, and calls the log stored in the log collection unit by real-time distributed streaming.

In addition, in the behavior-based rule processing apparatus according to the present invention, the real-time distributed streaming includes first normalization for classifying the loaded log according to data content or identifier, and the content of the log from the result of the first normalization. And a normalization process that performs second normalization, respectively, to check whether it corresponds to a block.

In addition, in the behavior-based rule processing apparatus according to the present invention, when the content of the log corresponds to the rule block, the real-time distributed streaming combines the block corresponding to the rule block and the block corresponding to the condition check block. Thus, the log action is output, and user mapping (identification of the user who caused the action) to the action is performed.

Further, in the behavior-based rule processing apparatus according to the present invention, the real-time distributed streaming and rule processing of the real-time rule processing unit are performed on a memory.

Further, in the behavior-based rule processing apparatus according to the present invention, the real-time distributed streaming distributes the normalization process in parallel, and the real-time rule processor multi-threads the behavior for each user or for each behavior. .

In addition, in the behavior-based rule processing apparatus according to the present invention, the real-time rule processing unit places a buffer on each user's behavior and processes the rule when processing the rule.

In addition, in the behavior-based rule processing apparatus according to the present invention, the real-time rule processing unit checks whether the behavior of the log matches the content of the rule block defined in advance and the metadata of the condition check block.

On the other hand, in order to achieve the above object, the behavior-based rule processing method according to the present invention includes a rule block for classifying the contents of logs collected by a log collection unit and used for rule processing according to a combination of predetermined values, and the In addition to including a condition check block for classifying predetermined metadata values of the log, rule processing is performed in real time by a real-time rule processing unit that pre-defines the behavior of the log, which is a combination of the rule block and the condition check block. As an action-based rule processing method of processing the log so that the log is processed, a first step (S100) in which the log collected by the log collection unit is called by real-time distributed streaming, and the log imported from the log collection unit is distributed in the real time. A second step (S200) of normalizing by a normalization process of streaming, and a third step of performing user identification (user mapping) of the log by mapping the log normalized by the real-time distributed streaming with the rule block (S300). ), and a fourth step (S400) of rule-processing the mapped log by the real-time rule processing unit.

In addition, in the behavior-based rule processing method according to the present invention, the normalization process includes: first normalization for classifying the imported log according to data content or separator, and the content of the log from the result of the first normalization Each of the second normalizations to check if it corresponds to is performed.

In addition, in the behavior-based rule processing method according to the present invention, when the content of the log corresponds to the rule block, the real-time distributed streaming combines a block corresponding to the rule block and a block corresponding to the condition check block. Thus, the log action is output, and the real-time rule processing unit checks whether the log action matches the content of the rule block defined in advance and metadata of the condition check block.

According to the present invention, logs generated in a large amount are defined as actions and processed in a standardized form, and the continuity of actions is defined as a rule to detect abnormal access, suitable for artificial intelligence learning, and capable of processing at high speed. There is.

1 is a system configuration diagram showing the overall configuration of an action-based rule processing apparatus according to the present invention.
2 is a diagram showing rule blocks and actions.
Fig. 3 is a diagram showing rule processing.
4 is a flow chart showing the processing flow of the behavior-based rule processing method according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the related drawings.

In the present invention, for ease of description, the Java language environment is described as an example, but the present invention is not limited thereto, and is applicable to all languages having an intermediate language such as C#.

1 is a system configuration diagram showing the overall configuration of an action-based rule processing apparatus according to the present invention.

Referring to FIG. 1, the behavior-based rule processing apparatus 1000 according to the present invention includes a log collection unit 100, a real-time rule processing unit 200, and a real-time distributed streaming 300.

Here, the log collection unit 100 collects logs used for rule processing from the plurality of agents 10. Also, it saves the collected logs.

The real-time rule processing unit 200 includes a rule block for classifying the contents of a log according to a combination of predetermined values, and a condition block for classifying predetermined metadata values of the log. The log behavior, which is a combination of a rule block and a condition check block, is predefined as a rule.

The real-time distributed streaming 300 serves to load the log stored in the log collection unit 100 in real time.

This will be described in more detail with reference to FIGS. 2 and 3.

2 is a diagram illustrating rule blocks and actions, and FIG. 3 is a diagram illustrating rule processing.

The types and types of logs vary greatly depending on the characteristics of the collection target or collection agent. Collection targets typically include a database system, a web server, a network packet, an operating system system, or an application, and logs left by such a collection target may be collected through a plurality of agents 10. Also, even if the collection target does not leave a log, the plurality of agents 10 may directly generate the log. For example, even if the application program does not leave a log, the agent 10 is installed in the application program to generate an application log necessary for analysis, or a case in which a log in the form of a file is left while monitoring packets of a network that does not leave a log may be used.

Logs collected through the plurality of agents 10 are normalized and sent to the real-time rule processing unit 200. Streaming processing may be required between the plurality of agents 10 and the real-time rule processing unit 200 for real-time performance of the analysis process. Real-time processing and distributed processing of a large amount of data (logs) are possible through streaming processing, and a streaming processing infrastructure may be configured by using the log collection unit 100 and the real-time distributed streaming 300.

For example, in the case of the log collection unit 100, it is a distributed messaging system for large-capacity log data, and serves as a buffer between the log agent and streaming data processing, and in the real-time distributed streaming 300, the log collection unit 100 Stored logs can be imported in a micro-batch method and data can be normalized according to business logic. Data (logs) subjected to normalization are transferred to the real-time rule processing unit 200 and used as input to business logic or rule processing process.

The contents of the log are usually strings recorded according to a specific format. Taking the most common web log as an example, a web log is a file that records information on when, where, and what pages were visited by users who visited a web site running on a specific web server or WAS. When looking at the contents of the web log, these information are recorded as strings according to the format.

Looking at the example of the web log in Table 1, the IP address of the client requested to the server, identification information, the user ID of the user found through HTTP authentication, the server request processing time, the HTTP method and HTTP used by the client. It contains the protocol, the request URL, the status code that the server sends to the client, and the size of the body that the server sends to the client. The entire log string can be analyzed through parsing in the analysis stage, but normally, it is normalized in advance according to the information required for analysis, and the result is used in the analysis stage. It is common to perform normalization according to the log data content or separator. As shown in Table 1, if the web log is normalized according to the delimiter (space) or the meaning of data, it can be normalized as shown in Table 2 below.

As shown in [Table 2], in order to analyze normalized data in real time (rule processing), and to create rules suitable for AI learning, we define "rule blocks" in advance, and all rules used in the rule processing engine are ruled. It is defined by using a rule block, which is a component of and a "condition check block" for additional condition check. In the end, if you see a log as an action, a rule means a pattern of these actions. Of course, a log can mean several actions according to a standard, but a log can be viewed as a user's action of accessing a web page in the case of a web log, and a user action such as a login or data inquiry in the case of a database log. Since the rule used in the analysis (rule processing) in the real-time rule processing unit 200 is a combination of these actions, all actions for rule generation are defined, and actions defined by all logs of the action-based rule processing unit 1000 If matched with, the rule processing process simply changes to an action comparison operation, and analysis can be performed without storing it in a storage medium such as a database before the analysis (rule processing) step.

Even if the format of the logs is fixed, if actions are classified according to the type of data content, the types of actions are enormously increased. Therefore, it is necessary to classify the actions by defining the type of action used in the rule, and this action type is a rule block, which is a component of the aforementioned rule. To view a log as an action means to see it as a predefined action when the content of the log is a certain value. The rule block, which is the type of action, is the criterion when classifying actions.

In FIG. 2, the type of action meaning "login attempt" is defined as a rule block with an ID of "try_login", and the value of web log data to be inspected is an E value representing the HTTP method requested by the client, and an access page. The value of F is the address, G is the HTTP protocol requested by the client, and H is the response code of the server. In the case of E="GET", F="/login.php", G="HTTP/1.1", H="200", it can be seen as an action meaning "login attempt". The condition values used to determine the behavior can of course differ depending on the user. This is because on other websites, the address of the login page may not be "/login.php", and there may be no login page.

In this way, only specific values of log data are compared to identify the type of action, and all values of the first normalization result are defined as meta data, and the rule block, which is the type of action, and meta data are combined to define an action. In order to match logs with actions (rule block + meta data), an additional secondary normalization process is required after the first normalization process based on data content and delimiter. In the second normalization process, it is checked whether the result of the first normalization of the log corresponds to the predefined behavior type (rule block). That is, in the behavior-based rule processing apparatus 1000 according to the present invention, the real-time distributed streaming 300 classifies (first normalizes) the log fetched from the log collection unit 100 according to data content or separator. Next, from the result of the first normalization, it is checked whether the log type corresponds to a rule block (second normalization). Thereafter, if it corresponds to a rule block, the ID of the rule block and the values of the result of the first normalization are added together, and the action, which is the result of the final normalization, comes out.

In Fig. 2, if E, F, G, H values meet the condition values of the rule block "try_login", the log is classified as "try_login", that is, the type of action (rule block) meaning "login attempt". do. However, it is not enough to distinguish between the types of actions alone. When there is a value of A, it is possible to know who made the "login attempt" action, and knowing the value of D can tell when the "login attempt" action was performed. Values other than these rule block condition check values are checked in the condition check block mentioned above. Therefore, all rules in the rule processing stage are composed of a combination of rule blocks and condition check blocks. If the rule "A user with IP 192.167.0.1 attempted to log in" is generated, the rule compares whether the rule block with the ID "try_login" and the A value is "192.167.0.1" as shown in FIG. 3. It consists of a condition check block. This is a rule block check called "try_login" to determine whether it is a "login attempt" action, and a condition check block to determine if the user with "192.167.0.1" is correct, and the user with IP 192.167.0.1 attempts to log in. This means that it can be automatically found by the real-time rule processing unit 200 through log analysis. That is, in the behavior-based rule processing apparatus 1000 according to the present invention, the real-time rule processing unit 200 checks whether the behavior of the log matches the content of the rule block defined in advance and the metadata of the condition check block.

If the rule is configured and the normalization result is defined as the values required for the rule block and condition check block, which are components of the rule, log data that comes in real time by the real-time distributed streaming 300 is stored in a simple rule block without database storage. Rule processing can be performed in-memory through ID check and comparison of specific item values of metadata, which is a result of primary normalization. That is, in the behavior-based rule processing apparatus 1000 according to the present invention, the real-time distributed streaming 300 and rule processing of the real-time rule processing unit 200 are performed on a memory.

Thereafter, in the behavior-based rule processing apparatus 1000 according to the present invention, when the content of the log corresponds to a rule block, the real-time distributed streaming 300 includes a block corresponding to the rule block and a block corresponding to the condition check block. Combined, the log behavior is output, and user mapping (identification of the user who caused the behavior) to the log behavior is performed.

In addition, there are cases where it is necessary to check consecutive behavior values (rule block ID value + meta data item value), such as the rule "User with IP 192.167.0.1 made two consecutive login attempts" In the case of the rule, "A user with 192.167.0.1 made login attempts consecutively within 10 seconds", the user with an IP 192.167.0.1 must wait 10 seconds after the first log in attempt to determine whether the rule passes. This is because if a user with an IP of 192.167.0.1 does not attempt to log in again within 10 seconds, the rule does not apply. In this way, there are cases in which it is necessary to look at the logs generated after a specific log when checking the rules. In this case, a buffer of actions is internally placed for rule processing to be processed. That is, in the behavior-based rule processing apparatus 1000 according to the present invention, the real-time rule processing unit 200 puts a buffer on each user's behavior when processing the rule and processes it.

In addition, in the real-time distributed streaming 300, parallel processing of the normalization process of logs is possible. Real-time distributed streaming 300 performs normalization processing, and action values resulting from normalization are input to the real-time rule processing unit 200 in parallel, and the real-time rule processing unit 200 processes multi-threading for each action. The rule processing performance can be improved by performing. In general, actions are classified based on users, and rules are also created based on users. If user identification is possible in the real-time distributed streaming stage or the analysis (rule processing) stage, rule processing can be processed in parallel for each user. That is, in the behavior-based rule processing apparatus 1000 according to the present invention, the real-time distributed streaming 300 distributes and processes the normalization process in parallel, and the real-time rule processing unit 200 multi-threads the behavior for each activity or user. .

On the other hand, if the log normalization result is stored in a database and then analyzed (rule processing), the input of the AI learning model becomes a query because the rule in the rule processing engine must be defined in the form of a query. Since the result of artificial intelligence learning is a rule in the form of SQL query, which is a kind of programming language, it is not suitable for AI that learns the rule and the rule processing result and uses it for rule enhancement. However, if the rules are configured using simple blocks, the pattern of the rules can be used as input and output (output) of the artificial intelligence learning model, making it easy to introduce artificial intelligence in the future. That is, if a combination of rule blocks already registered in the rule engine is generated as a result of learning, the advancement of an existing rule (optimum combination of rule blocks) can be generated as a learning result.

Next, Figure 4 is a flow chart showing the processing flow of the behavior-based rule processing method according to the present invention.

Referring to FIG. 4, the method for processing an action-based rule according to the present invention includes a rule block that is collected by the log collection unit 100 and classifies the contents of a log used for rule processing according to a combination of predetermined values, and a log The rule processing is performed in real time by the real-time rule processing unit 200 that includes a condition check block that classifies the predetermined metadata values of the rule block and the log behavior, which is a combination of the condition check block, as rules. It is an action-based rule processing method that processes logs and proceeds in a total of 4 steps.

In the first step (S100), the real-time distributed streaming 300 calls the log collected by the log collection unit 100.

In the second step (S200), the log retrieved from the log collection unit 100 is normalized by the normalization process of the real-time distributed streaming 300.

In the third step (S300), the log normalized by the real-time distributed streaming 300 is mapped with the rule block to perform user identification (user mapping) of the log.

In the fourth step (S400). The mapped log is rule-processed by the real-time rule processing unit 200.

In this case, in the normalization process, a first normalization for classifying the loaded log according to data content or a delimiter and a second normalization for checking whether the type of log corresponds to a rule block from the result of the first normalization are respectively performed.

On the other hand, when the type of log corresponds to the rule block, the real-time distributed streaming 300 combines the block corresponding to the rule block and the block corresponding to the condition check block to output the behavior of the log, and the real-time rule processing unit 200 ) Checks whether the log behavior matches the content of the predefined rule block and the metadata of the condition check block.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains will be able to make various modifications and variations without departing from the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

10: multiple agents
100: log collection unit
200: Real-time rule processing unit
300: Real-time distributed streaming
1000: behavior-based rule processing device

Claims (10)

A log collection unit that collects logs used for rule processing from a plurality of agents and stores the collected logs,
The rule block includes a rule block for classifying the contents of the log according to a combination of predetermined values, and a condition block for classifying predetermined metadata values of the log, and the rule block, A real-time rule processing unit that pre-defines the actions of the log, which is a combination of the condition check blocks, as rules,
The log stored in the log collection unit is called by real-time distributed streaming,
The real-time distributed streaming,
A first normalization for classifying the loaded log according to data content or separator,
And a normalization process of performing second normalization, respectively, for checking whether the content of the log corresponds to the rule block from the result of the first normalization,
If the contents of the log correspond to the rule block,
The real-time distributed streaming combines a block corresponding to the rule block and a block corresponding to the condition check block, outputs the behavior of the log, and performs user mapping (identifying the user who caused the behavior) to the behavior. And
The real-time distributed streaming and rule processing of the real-time rule processor are performed on a memory. delete delete delete The method of claim 1,
In the real-time distributed streaming, the normalization process is distributed in parallel,
The real-time rule processing unit multi-threading the action for each user or for each action. The method of claim 1,
The real-time rule processing unit puts a buffer on each user's actions when processing the rules and processes them. The method of claim 1,
The real-time rule processing unit checks whether the action of the log matches the content of the rule block defined in advance and the metadata of the condition check block. The rule includes a rule block that is collected by the log collection unit and classifies the contents of the log used for rule processing according to a combination of predetermined values, and a condition check block that classifies predetermined metadata values of the log. As an action-based rule processing method for processing the log so that rule processing is performed in real time by a real-time rule processing unit that pre-defines the actions of the log, which is a combination of a block and the condition check block, as a rule,
A first step (S100) in which real-time distributed streaming calls the logs collected by the log collection unit,
A second step (S200) of normalizing the log retrieved from the log collection unit by a normalization process of the real-time distributed streaming,
A third step (S300) of performing user identification (user mapping) of the log by mapping the log normalized by the real-time distributed streaming with the rule block,
A fourth step (S400) of rule-processing the mapped log by the real-time rule processing unit,
The normalization process,
A first normalization for classifying the loaded log according to data content or separator,
From the result of the first normalization, each of the second normalization to check whether the content of the log corresponds to the rule block,
If the contents of the log correspond to the rule block,
The real-time distributed streaming combines a block corresponding to the rule block and a block corresponding to the condition check block to output the behavior of the log,
The real-time rule processing unit checks whether the action of the log matches the content of the rule block defined in advance and the metadata of the condition check block,
The real-time distributed streaming and rule processing of the real-time rule processor are performed on a memory.
delete delete

Images