KR20200066428A - A unit and method for processing rule based action - Google Patents

Abstract

An objective of the present invention is to provide an action-based rule processing apparatus and a processing method thereof which can define logs generated in a large amount as an action, process the logs in a standardized form, define continuity of actions as a rule to detect abnormal access, be suitable for artificial intelligence learning, and process the actions at high speeds. To achieve the objective, the action-based rule processing apparatus according to the present invention includes a log collection unit which collects logs used for rule processing from a plurality of agents and stores the collected logs; a rule block which classifies contents of the log according to a combination of predetermined values; and a real-time rule processing unit which includes a condition block for classifying predetermined metadata values of the log and pre-defining actions of the log, which is a combination of the rule block and the condition check block as a rule, wherein the log stored in the log collection unit is retrieved by real-time distribution streaming.

Description

A behavior-based rule processing device and a processing method thereof {A UNIT AND METHOD FOR PROCESSING RULE BASED ACTION}

The present invention relates to a behavior-based rule processing apparatus and a processing method thereof, and more particularly, to a behavior-based rule processing apparatus and a processing method suitable for artificial intelligence learning and real-time analysis.

Recently, due to the rapid growth of hardware and the expansion of services provided through the web, the amount of data used in the Internet network is rapidly increasing. Accordingly, the abnormal access for the purpose of attack is increasing explosively as well as the access of the user who normally uses the service.

Existing analysis systems used to detect abnormal access have determined threats by extracting abnormal information found in specific actions, that is, logs, based on rules specified by the administrator. However, in the traditional log analysis system, it is unable to cope with the explosively increasing amount of log data, or it has limitations that it cannot respond to increasingly intelligent attack patterns.

As a method to overcome the limitations of large-capacity processing, most analysis systems adopt a method that enhances hardware performance, but there are still limitations such as bandwidth limitations and IO (input/output) delays, and it is an intelligent and difficult to judge approach. Research has been conducted to extract patterns through an artificial intelligence model to detect, but most of the rule definition methods used in rule engines that process existing rules use a programming method, so they can be used in artificial intelligence models. There was a problem that can not be formalized in form.

For example, in the general analysis step (rule engine), the normalization log is first stored in a storage medium such as a database, and the stored information is loaded to perform analysis logic (rule processing). However, even if a real-time streaming system is used, this method is not suitable for real-time processing because it requires a process of storing streaming results in a database before analysis (rule processing) and processing time for querying the database in the analysis step.

In addition, when the log types are various, the normalized result items of the log are specified as columns of each table and stored in a table in the database. In the analysis logic, processing that takes a long time, such as a JOIN query for multiple tables, takes a long time. It is not suitable for real-time analysis because it is necessary.

In addition, rules in the rule processing engine must be defined in the form of queries (SQL Query), and are not suitable for AI learning models for rule generation and rule enhancement. Because the result of AI learning is a rule in the form of SQL query, which is a kind of programming language, it is necessary to implement a learning model close to natural language learning.

Accordingly, the present invention has been proposed to meet the above-described needs, and its purpose is to define logs that occur in large quantities as actions and process them in a standardized form, and define continuity of actions as rules to detect abnormal access, It is to provide an action-based rule processing device and a processing method suitable for artificial intelligence learning and capable of processing at a high speed.

In order to achieve the above object, the behavior-based rule processing apparatus according to the present invention collects logs used for rule processing from a plurality of agents and a log collection unit for storing the collected logs, and the contents of the logs A rule block that classifies according to a combination of predetermined values, and a condition block that classifies predetermined metadata values of the log, and includes the rule block and the condition check block. It includes a real-time rule processing unit that pre-defines the actions of the log as a rule, and retrieves the log stored in the log collection unit by real-time distributed streaming.

In addition, in the behavior-based rule processing apparatus according to the present invention, in the real-time distributed streaming, the first normalization classifying the loaded log according to data content or delimiter, and the log contents from the result of the first normalization are the rules And a normalization process, each of which performs a second normalization to check whether it corresponds to the block.

In addition, in the behavior-based rule processing apparatus according to the present invention, when the content of the log corresponds to the rule block, the real-time distributed streaming combines the block corresponding to the rule block and the block corresponding to the condition check block To output the log's action, and perform user mapping (identifying the user who generated the action) to the action.

In addition, in the behavior-based rule processing apparatus according to the present invention, the real-time distributed streaming and the rule processing of the real-time rule processing unit are performed on a memory.

In addition, in the behavior-based rule processing apparatus according to the present invention, the real-time distributed streaming distributes the normalization process in parallel, and the real-time rule processing unit multi-threads the behavior for each user or each activity. .

In addition, in the behavior-based rule processing apparatus according to the present invention, the real-time rule processing unit processes the rule with a buffer in user-specific actions.

In addition, in the behavior-based rule processing apparatus according to the present invention, the real-time rule processing unit checks whether the behavior of the log matches the content of the rule block defined in advance and the metadata of the condition check block.

On the other hand, in order to achieve the above object, the behavior-based rule processing method according to the present invention includes a rule block that classifies contents of a log collected by a log collection unit and used for rule processing according to a combination of predetermined values, and In addition to including a condition check block that classifies predetermined metadata values of a log, real-time rule processing is performed by a real-time rule processing unit that pre-defines rules of the log as a rule that is a combination of the rule block and the condition check block. As a behavior-based rule processing method of processing the log to be lost, a first step (S100) in which real-time distributed streaming calls the log collected by the log collection unit, and the real-time distribution of the log retrieved from the log collection unit The second step (S200) of normalizing by the normalization process of streaming, and the third step of performing user identification (user mapping) of the log by mapping the log normalized by the real-time distributed streaming with the rule block (S300) ) And a fourth step of processing the mapped log by the real-time rule processing unit (S400 ).

In addition, in the behavior-based rule processing method according to the present invention, the normalization process includes: a first normalization classifying the loaded log according to data content or a separator, and the log content from the result of the first normalization is the rule block Each of the second normalization is performed to check whether it corresponds to.

Further, in the behavior-based rule processing method according to the present invention, when the content of the log corresponds to the rule block, the real-time distributed streaming combines a block corresponding to the rule block and a block corresponding to the condition check block The log action is output, and the real-time rule processing unit checks whether the log action matches the content of the rule block defined in advance and the metadata of the condition check block.

According to the present invention, logs generated in a large amount are defined as actions and processed in a standardized form, and continuity of actions is defined as rules to detect abnormal access, suitable for artificial intelligence learning, and capable of processing at high speed. There is.

1 is a system configuration diagram showing the overall configuration of a behavior-based rule processing apparatus according to the present invention.
2 is a diagram showing rule blocks and actions.
3 is a diagram showing rule processing.
Figure 4 is a floor chart showing the processing flow of the behavior-based rule processing method according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to related drawings.

For ease of explanation, the present invention describes the Java language environment as an example, but is not limited thereto, and is applicable to all languages having an intermediate language such as C#.

1 is a system configuration diagram showing the overall configuration of a behavior-based rule processing apparatus according to the present invention.

1, the behavior-based rule processing apparatus 1000 according to the present invention includes a log collection unit 100, a real-time rule processing unit 200, and a real-time distributed streaming 300.

Here, the log collection unit 100 collects logs used for rule processing from a plurality of agents 10. Also, it serves to store the collected log.

The real-time rule processing unit 200 includes a rule block that classifies the log contents according to a combination of predetermined values, and a condition block that classifies predetermined metadata values of the log. The behavior of the log, which is a combination of the rule block and the condition check block, is defined in advance as a rule.

The real-time distributed streaming 300 serves to fetch the log stored in the log collection unit 100 in real time.

This will be described in more detail with reference to FIGS. 2 and 3.

2 is a diagram showing rule blocks and actions, and FIG. 3 is a diagram showing rule processing.

The types and types of logs vary greatly depending on the characteristics of the collection target or collection agent. The target of collection includes a database system, a web server, a network packet, an operating system, or an application, and the logs left by the collection target can be collected through a plurality of agents 10. Further, even if the collection target does not leave a log, the plurality of agents 10 may directly generate the log. For example, even if the log is not left in the application program, the agent 10 is mounted in the application program to generate an application log required for analysis or to log in a file form while monitoring a packet in a network where no log is left.

Logs collected through the plurality of agents 10 are normalized and sent to the real-time rule processing unit 200. Streaming processing may be required between the plurality of agents 10 and the real-time rule processing unit 200 for real-time analysis. Real-time processing and distributed processing of large amounts of data (logs) are possible through streaming processing, and a streaming processing infrastructure can be configured by utilizing the log collection unit 100 and real-time distributed streaming 300.

For example, in the case of the log collection unit 100, a distributed messaging system for large-capacity log data serves as a buffer between the log agent and the streaming data processing, and in the real-time distributed streaming 300, the log collection unit 100 The stored logs can be imported in a micro-batch method to perform data normalization processing according to business logic. The normalized data (logs) are transferred to the real-time rule processing unit 200 and used as input to business logic or rule processing.

The contents of the log are usually strings recorded according to a specific format. Taking the most common web log as an example, the web log is a file that records information about when, where, and what pages were visited by users who visited a website running on a specific web server or WAS. If you look at the contents of the web log, these information is recorded as a string according to the format.

Looking at the example of the web log in Table 1, the IP address of the client requesting the server, the identity information, the user ID of the user found through HTTP authentication, the server request processing time, the HTTP method and HTTP used by the client It contains the protocol, the request URL, the status code the server sends to the client, and the size of the body the server sends to the client. Although the entire log string can be analyzed through parsing in the analysis step, it is normally normalized beforehand according to the information required for analysis, and the result is used in the analysis step. It is common to normalize according to the log data content or delimiter. As shown in Table 1, if the web log is normalized according to the separator (space) or the meaning of the data, it can be normalized as shown in Table 2 below.

As shown in [Table 2], in order to analyze the normalized data more in real time (rule processing) and to create a rule suitable for AI learning, the rule rule component "rule block" is defined in advance, and all rules used in the rule processing engine are ruled. It is defined by using the rule block, which is a component of and the "condition check block" for additional condition check. A rule means a pattern of these actions if the log is viewed as an action. Of course, the log can mean several actions depending on the criteria, but the log can be viewed as a user's action such as a user's access to a web page in the case of a web log, or a login or data inquiry in the case of a database log. Since the rules used in the analysis (rule processing) in the real-time rule processing unit 200 are combinations of these actions, all actions for rule generation are defined, and actions defined by all logs of the action-based rule processing apparatus 1000 If matched with, the rule processing process is simply changed to a behavior comparison operation, and analysis can be performed without storing in a storage medium such as a database before the analysis (rule processing) step.

Even if the format of the logs is determined, if the actions are classified according to the type of data content, the types of the actions are greatly increased. Therefore, actions must be classified by defining the type of action used in the rule, and this action type is a rule block that is a component of the aforementioned rule. Viewing a log as an action means that when the content of the log is a certain value, it is viewed as a predefined action. A rule block that is a type of action is a criterion when classifying actions.

In FIG. 2, the type of the action having the meaning of “login attempt” is defined as a rule block having an ID of “try_login”. At this time, the value of the web log data to be checked is the E value indicating the HTTP method requested by the client and the access page. The F value for the address, the client's request HTTP protocol G, and the server's response code H. In the case of E="GET", F="/login.php", G="HTTP/1.1", and H="200", it can be considered as an action of "login attempt". Condition values used for discrimination of actions may of course vary depending on the user. This is because the address of the login page may not be "/login.php" on other websites, or the login page may not exist.

As described above, the behavior type is classified by comparing only specific values among the contents of log data, and all values of the primary normalization result are defined as metadata, and the rule block, which is the type of behavior, and metadata are combined to define the behavior. In order to match the logs with actions (rule block + meta data), a second normalization process is required after the first normalization process based on the content and delimiters of the data. In the second normalization process, it is checked whether the first normalization result of the log corresponds to a predefined type of action (rule block). That is, in the behavior-based rule processing apparatus 1000 according to the present invention, the real-time distributed streaming 300 classifies the logs retrieved from the log collection unit 100 according to data contents or delimiters (first normalization). Next, it is checked whether the log type corresponds to the rule block from the result of the first normalization (second normalization). Thereafter, if the rule block corresponds, the ID of the corresponding rule block and the primary normalization result values are combined to produce an action that is the final normalization result.

In FIG. 2, if the E, F, G, and H values meet the condition values of the rule block “try_login”, the corresponding log is classified as a type of action (rule block) having the meaning of “try_login”, that is, “try to log in”. do. However, this kind of action alone is not enough to distinguish the action. The A value is needed to know who has done the "login attempt" and know the D value to know when the "login attempt" was done. Values other than these rule block condition check values are checked in the condition check block mentioned above. Therefore, all rules in the rule processing step are composed of a combination of rule blocks and condition check blocks. If the rule "User with IP 192.167.0.1 tried to log in" is created, the rule compares the rule block with ID "try_login" and the value of A is "192.167.0.1" as shown in FIG. It consists of a condition check block. This determines whether or not it is an "try to log in" action through a rule block check called "try_login", and determines whether a user with "192.167.0.1" is correct through a condition check block when a user with an IP of 192.167.0.1 tries to log in. This means that it can be automatically found in the real-time rule processing unit 200 through log analysis. That is, in the action-based rule processing apparatus 1000 according to the present invention, the real-time rule processing unit 200 checks whether a log action matches the content of the rule block defined in advance and the metadata of the condition check block.

In this way, if the rule is configured and the normalization result is defined with the values required for the rule block, which is a component of the rule, and the condition check block, the log data coming in real time by the real-time distributed streaming 300 is stored in a simple rule block without database storage. Rule processing can be performed in-memory through ID check and comparison of specific item values of meta data, which is the result of primary normalization. That is, in the behavior-based rule processing apparatus 1000 according to the present invention, the real-time distributed streaming 300 and the rule processing of the real-time rule processing unit 200 are performed on the memory.

Thereafter, in the behavior-based rule processing apparatus 1000 according to the present invention, when the contents of the log correspond to the rule block, the real-time distributed streaming 300 blocks the block corresponding to the rule block and the block corresponding to the condition check block Combined, the log actions are output, and user mapping (identifying the user who caused the action) to the log actions is performed.

In addition, in some cases, it is necessary to check a series of action values (rule block ID value + metadata item value), such as the rule "A user with an IP of 192.167.0.1 made two consecutive login attempts." In the case of the rule that a user with 192.167.0.1 attempted a login within 10 seconds consecutively, a user with an IP of 192.167.0.1 must wait 10 seconds after the first log in attempting to log in to determine whether or not the rule has passed. If the user with IP 192.167.0.1 does not try to log in again within 10 seconds, the rule is not applied. In this case, it is necessary to check the logs generated after a specific log when checking rules. In this case, a buffer of actions is placed internally for rule processing. That is, in the action-based rule processing apparatus 1000 according to the present invention, the real-time rule processing unit 200 processes a rule with a buffer in user-specific actions.

Also, in the real-time distributed streaming 300, parallel processing of the normalization process of logs is possible. The real-time distributed streaming 300 performs normalization processing, and the action values resulting from the normalization processing are input to the real-time rule processing unit 200 in parallel, and the real-time rule processing unit 200 processes multi-threading for each action. By performing, rule processing performance can be improved. Usually, actions are classified based on users, and rules are also created based on users. If the user can be identified in the real-time distributed streaming step or the analysis (rule processing) step, the rule processing can be processed in parallel for each user. That is, in the action-based rule processing apparatus 1000 according to the present invention, the real-time distributed streaming 300 distributes the normalization process in parallel, and the real-time rule processing unit 200 performs multi-threading of the actions by action or by user. .

On the other hand, if analysis (rule processing) is performed after storing the normalized result of the log in the database, the rules in the rule processing engine must be defined in the form of a query (SQL Query), so the input of the AI learning model becomes a query, Because the result of AI learning is a rule in the form of SQL query, which is a kind of programming language, it is not suitable for AI that learns rules and results of rule processing and utilizes them to advance rules. However, if the rules are constructed using simple blocks, the rule patterns can be used as input and output (results) of the AI learning model, making it easy to introduce artificial intelligence later. That is, if a combination of rule blocks already registered in the rule engine is generated as a learning result, an upgrade of the existing rule (optimal rule block combination) can be generated as a learning result.

Next, Figure 4 is a floor chart showing the processing flow of the behavior-based rule processing method according to the present invention.

Referring to FIG. 4, the behavior-based rule processing method according to the present invention includes a rule block that classifies the contents of a log collected by the log collection unit 100 and used for rule processing according to a combination of predetermined values, and a log In addition to including a condition check block that classifies predetermined metadata values of the rule block, rule processing is performed in real time by a real-time rule processing unit 200 that pre-defines the log blocks, which are combinations of the rule block and the condition check block, as rules. It is a process-based rule processing method for processing logs, and proceeds in four stages.

In the first step (S100), the real-time distributed streaming 300 loads the logs collected by the log collection unit 100.

In the second step (S200 ), the log retrieved from the log collection unit 100 is normalized by the normalization process of the real-time distributed streaming 300.

In the third step (S300), the log normalized by the real-time distributed streaming 300 is mapped to the rule block to perform user identification (user mapping) of the log.

In the fourth step (S400). The mapped log is rule-processed by the real-time rule processing unit 200.

At this time, the normalization process performs first normalization to classify the loaded log according to data content or separator, and second normalization to check whether the log type corresponds to a rule block from the results of the first normalization.

On the other hand, if the type of log corresponds to a rule block, the real-time distributed streaming 300 combines the block corresponding to the rule block and the block corresponding to the condition check block to output the log behavior, and the real-time rule processing unit 200 ) Checks whether the behavior of the log matches the contents of the predefined rule block and the metadata of the condition check block.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains may make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical spirits within the equivalent range should be interpreted as being included in the scope of the present invention.

10: multiple agents
100: log collection unit
200: real-time rule processing unit
300: Real-time distributed streaming
1000: Behavior-based rule processing device

Claims (10)

A log collection unit that collects logs used for rule processing from a plurality of agents and stores the collected logs;
The rule block includes a rule block for classifying contents of the log according to a combination of predetermined values, and a condition block for classifying predetermined metadata values of the log, and the rule block, It includes a real-time rule processing unit that pre-defines the actions of the log as a combination of the condition check block,
A behavior-based rule processing device for loading the log stored in the log collection unit by real-time distributed streaming. According to claim 1,
The real-time distributed streaming,
A first normalization to classify the loaded log according to data content or separator,
And a normalization process, each of which performs a second normalization to check whether the log contents correspond to the rule block from the result of the first normalization. According to claim 2,
When the content of the log corresponds to the rule block,
In the real-time distributed streaming, a block corresponding to the rule block and a block corresponding to the condition check block are combined to output the log action, and user mapping (identifying the user who caused the action) is performed. An action-based rule processing device. The method of claim 3,
The real-time distributed streaming and the rule processing of the real-time rule processing unit are an action-based rule processing device performed on a memory. The method of claim 3,
The real-time distributed streaming distributes the normalization process in parallel,
The real-time rule processing unit is an action-based rule processing device that performs multi-threading processing of the actions for each user or each action. The method of claim 3,
The real-time rule processing unit is a behavior-based rule processing apparatus that processes a rule by placing a buffer in user-specific actions. The method of claim 3,
The real-time rule processing unit is an action-based rule processing device that checks whether the action of the log matches the content of the rule block defined in advance and the metadata of the condition check block. A rule block for classifying contents of a log collected by a log collection unit and used for rule processing according to a combination of predetermined values, and a condition check block for classifying predetermined metadata values of the log, and the rule As an action-based rule processing method of processing the log so that rule processing is performed in real time by a real-time rule processing unit that pre-defines actions of the log, which is a combination of a block and the condition check block, as a rule,
A first step (S100) in which real-time distributed streaming loads the log collected by the log collection unit;
A second step (S200) of normalizing the log retrieved from the log collection unit by a normalization process of the real-time distributed streaming,
A third step (S300) of mapping the log normalized by the real-time distributed streaming with the rule block to perform user identification (user mapping) of the log;
And a fourth step (S400) of processing the mapped log by the real-time rule processing unit (S400). The method of claim 8,
The normalization process,
A first normalization to classify the loaded log according to data content or separator,
A behavior-based rule processing method for performing a second normalization, each of which checks whether the log content corresponds to the rule block, from the result of the first normalization. The method of claim 8,
When the content of the log corresponds to the rule block,
The real-time distributed streaming combines the block corresponding to the rule block and the block corresponding to the condition check block and outputs the behavior of the log,
The real-time rule processing unit is an action-based rule processing method for checking whether the log action matches the content of the rule block, which is predefined, and metadata of the condition check block.

Images