KR102656541B1 - Device, method and program that analyzes large log data using a distributed method for each log type - Google Patents

Abstract

The present invention relates to a data analysis and processing device for each log type for large-capacity event logs, which parses large-capacity log data received from security equipment according to log type classification criteria and classifies the parsed log data according to the log type classification criteria. By analyzing based on a ruleset, there is the effect of parallel and distributed processing of log data received from security equipment.

Description

Device, method and program for analyzing large log data using a distributed method for each log type {Device, method and program that analyzes large log data using a distributed method for each log type}

The present invention relates to a log data analysis device, and more specifically, to a device for analyzing large amounts of log data using a distribution method for each log type.

SIEM (Security Information Event Management) or CEP (Complex Event) analyzes event log data generated from various devices in real time to determine whether there are threats from internal or external sources.

However, it is possible to process all log data from reception to analysis by installing additional SIEM / CEP modules on all server equipment and allocating resources separately within each server. However, in reality, the burden on the servers becomes very large and resource constraints are required. There are many cases where smooth operation is impossible due to reasons such as shortage.

In addition, when processing analysis within each server, the analyzed results are very fragmented, so it is also difficult to connect with the analyzed results from other servers to obtain continuous results of all equipment.

Therefore, if you prepare a separate SIEM / CEP server and collect all event logs into one and process them in batches, you can obtain overall and continuous analysis results from log data collected from many devices.

In addition, conventionally, various types of log data received from various security devices were analyzed in batches, but recently, as the amount of data has increased rapidly, this analysis method has become unable to process all log data.

Accordingly, there is a need for technology to analyze and process large amounts of log data in a different way than before, but such technology is not currently available to the public.

Public Patent Publication No. 10-2018-0061891, (2018.06.08)

The present invention to solve the problems described above parses large amounts of log data received from security equipment according to log type classification criteria, and analyzes the parsed log data based on the rules set for each log type classification standard to detect threats. We want to determine whether it is information or not.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

An apparatus for analyzing and processing data by log type for an event log according to an embodiment of the present invention to solve the above-described problem includes: a communication unit that communicates with a plurality of different security devices; And when a large amount of log data larger than a preset size is received in real time from the plurality of different security devices through the communication unit, the received log data is parsed according to a preset log type classification standard, and the parsed log data is Threat information is determined by analysis based on the rule set set for each log type classification standard, and the received log data is distributed to the plurality of nodes based on the amount of data classified and accumulated in the plurality of nodes during a preset period. It includes a processor that sets the log type classification criteria so that they are uniformly distributed.

In addition, it further includes a cluster consisting of a plurality of nodes, wherein the plurality of nodes are individually equipped with an analysis application corresponding to a log type set for each node, and the processor is provided for each node. By executing the analysis application, log data received from the plurality of different security devices can be processed in parallel and distributed.

In addition, the rule set includes at least one rule for determining whether log data received from the plurality of security devices corresponds to threat information, and a separate analysis module is assigned to each according to the type of log data, and the The processor can analyze the log data using a rule set set according to each log type.

Additionally, if the proportion of log data for a specific log type occupies a preset level or more, the processor may expand the log type classification criteria for the corresponding node to a plurality of log types.

In addition, the classification criteria for each log type are determined based on the plurality of different types of security equipment, and the processor classifies the received log data when the proportion of log data for a specific log type occupies a preset level or more. Security equipment classification for each node can be rearranged so that it is distributed.

In addition, the data analysis processing device further includes at least one database classified according to a period for which the parsed log data is stored, and the processor stores the parsed log data in the at least one database, and stores the parsed log data in the at least one database, and stores the parsed log data in the at least one database. The log data can be deleted after a preset storage period.

In addition, it further includes an analysis module for analyzing past log data, and the processor can generate past data analysis results by analyzing a corresponding database based on target equipment and target period for which past log data analysis is required.

In addition, the processor generates an alarm signal when the log data corresponds to threat information as a result of the determination, and checks the ruleset for detecting threat information when the number of alarm signals generated during a preset time exceeds a threshold range. A request signal can be generated.

In addition, the data analysis and processing method for each log type for the event log according to an embodiment of the present invention to solve the above-described problem is a method performed by a data analysis processing device, and is a method of analyzing real-time logs from a plurality of different security devices. receiving data; parsing the received log data according to preset log type classification criteria; Analyzing the parsed log data based on a rule set set for each log type classification standard to determine whether it is threat information; and setting the log type classification criteria so that the received log data is uniformly distributed to the plurality of nodes based on the amount of data classified and accumulated in the plurality of nodes during a preset period.

In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium recording a computer program for executing the method may be further provided.

According to the present invention as described above, a large amount of log data received from security equipment is parsed according to log type classification criteria, and the parsed log data is analyzed based on a rule set set for each log type classification standard, so that the log data received from security equipment This has the effect of allowing parallel and distributed processing of log data.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a block diagram of a data analysis processing device for each log type for an event log according to an embodiment of the present invention.
Figure 2 is a flowchart of a data analysis processing method for each log type for an event log according to an embodiment of the present invention.
Figure 3 is a diagram illustrating the configuration of a cluster in an embodiment of the present invention.
Figure 4 is a diagram illustrating a data analysis processing method for each log type for a conventional event log.
Figure 5 is a diagram illustrating a data analysis processing method for each log type for an event log according to an embodiment of the present invention.
Figure 6 is a diagram illustrating the configuration of a data analysis processing device for each log type for an event log according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide a general understanding of the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements. Although “first”, “second”, etc. are used to describe various components, these components are of course not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may also be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

1 is a block diagram of a data analysis processing device for each log type for an event log according to an embodiment of the present invention.

Figure 2 is a flowchart of a data analysis processing method for each log type for an event log according to an embodiment of the present invention.

Figure 3 is a diagram illustrating the configuration of a cluster in an embodiment of the present invention.

Figure 4 is a diagram illustrating a data analysis processing method for each log type for a conventional event log.

Figure 5 is a diagram illustrating a data analysis processing method for each log type for an event log according to an embodiment of the present invention.

Figure 6 is a diagram illustrating the configuration of a data analysis processing device for each log type for an event log according to an embodiment of the present invention.

A cluster (or computer cluster) is a concept that connects multiple computers through a network so that they can be used as one computer. Computer clusters are used in many fields, including computer operating systems, computer hardware, and statistical data. The components of a computer cluster are typically connected by a high-speed local area network.

Clusters typically boast better performance and reliability than a single computer, and are much more efficient than a single computer.

Referring to FIG. 1, the data analysis and processing device 100 for each log type for an event log according to an embodiment of the present invention includes a processor 110, a communication unit 120, a storage unit 130, an input/output unit 140, It includes a parsing unit 150, an analysis unit 160, a search unit 170, and a log receiving module.

However, in some embodiments, the data analysis processing device 100 may include fewer or more components than those shown in FIG. 1 .

The communication unit 120 communicates with a plurality of different security devices, field manager terminals, etc., and specifically receives real-time log data from a plurality of different security devices, and the processor 110 generates a threat information discovery signal for the log data. Then, the signal can be transmitted to the field manager terminal.

The communication unit 120 may further include a log receiving module 190 that is responsible only for receiving logs.

The storage unit 130 includes at least one storage means, and for example, log data parsed and classified by the parsing unit 150 is stored.

In addition, the storage unit 130 can store various commands and algorithms for executing data analysis processing methods, and in addition, various artificial intelligence models can be stored together.

The storage unit 130 may further include a log type storage unit in which the log type is stored.

The input/output unit 140 may be composed of separate input means and output means, and can receive various control signals from the manager or field manager through the input means, and determine the presence or absence of threat information through the output means, resulting in an error. Messages, etc. can be output.

The parsing unit 150 parses log data received through the log receiving module according to preset log type classification criteria.

The parsing unit 150 may include at least one parsing module, each of which may include a separate parsing module depending on the log type, so that each parsing module may parse log data according to log type classification criteria.

In an embodiment of the present invention, the parsing unit 150 is illustrated as parsing the received log data according to the log type classification criteria. However, the log type classification criteria is only the top consideration in the parsing of the parsing unit 150, and the parsing/ The classification criteria are not limited to one log type classification criteria.

For example, further classification criteria such as SQL query where, group by having, etc. can be applied.

The analysis unit 160 may include at least one analysis module, each of which includes a separate analysis module depending on the log type, and each analysis module parses and classifies the log type from the parsing unit 150. Analysis can be performed on log data.

At this time, each analysis module is equipped with an analysis application, and log data can be analyzed using a rule set optimized for each log type.

The search unit 170 can execute a search function for log data stored in the storage unit 130 and includes at least one search module, specifically, a first search module for integrated analysis of past data and a real-time search function. It may include a second search module for .

In one embodiment, a search module may be included in the analysis unit 160 for real-time search.

The processor 110 is responsible for controlling the components within the data analysis processing device 100, and processes data analysis according to an embodiment of the present invention by executing/using instructions, algorithms, artificial intelligence models, etc. stored in the storage unit 130. method can be implemented.

With reference to FIG. 2, the process of the data analysis processing method according to an embodiment of the present invention will be described in detail.

The processor 110 receives real-time log data from a plurality of different security devices through the communication unit 120. (S100)

Specifically, a plurality of different security devices received through the communication unit 120 are subject to analysis processing, and as shown in FIGS. 4 and 5, various security devices operated by clients such as IDS, IPS, WAF, FW, WEB, etc. are applied. It is possible, and other security equipment can be bundled with ETC.

The processor 110 parses the log data received from S100 according to preset log type classification criteria. (S200)

In one embodiment, as described above, the parsing unit 150 may include separate parsing modules depending on the log type, and each parsing module may classify the log data received through the communication unit 120 according to the log type classification criteria. It can be parsed accordingly.

Then, the processor 110 stores the log data parsed and classified through the parsing unit 150 in the storage unit 130 according to the classification result.

In detail, the processor 110 stores log data in the storage unit 130 and stores it as classified through the parsing unit 150 to facilitate real-time analysis and future past data search and analysis.

At this time, the storage unit 130 includes at least one database classified according to the period for which log data classified through the parsing unit 150 is stored.

For example, the storage unit 130 may include a first database that stores log data for 1 week, a second database that stores log data for 1 month, and a third database that stores log data for 1 year. .

The processor 110 stores the parsed log data in at least one database and deletes the log data after a preset storage period from the point of storage.

The processor 110 analyzes the log data parsed in S200 based on the rule set set for each log type classification standard to determine whether it is threat information. (S300)

In an embodiment of the present invention, the rule set stores at least one rule for determining whether log data corresponds to threat information, and these rules can be set by security control personnel or personnel.

In addition, when specific log data corresponds to threat information through S300, the processor 110 can generate a warning to the input/output unit 140, a security control device, and a terminal so that a security control agent/person in charge can confirm it.

The data analysis processing device 100 according to an embodiment of the present invention includes a cluster consisting of a plurality of nodes.

Each node is individually equipped with an analysis application corresponding to the log type set for each node.

The processor 110 can parallel and distribute large amounts of log data received from a plurality of different security devices by executing an analysis application provided for each node.

In an embodiment of the present invention, classification criteria for each log type may be determined based on a plurality of different types of security equipment.

Figure 4 is a diagram illustrating a data analysis processing method for each log type for a conventional event log.

In an embodiment of the present invention, there are five types of security equipment operated by a client, and accordingly, there are five types of log types.

Referring to FIG. 4, conventionally, when 100 pieces of log data are received, when analyzing and processing them, calculations are performed on 5 types of log data and 100 rules for all 100 pieces of log data, and 100 x 5 x 100 = 50,000 calculations were performed.

However, when applying the data analysis treatment device according to the embodiment of the present invention applying the above-described configurations, 100 log data received from the security equipment are divided into 5 log types according to the 5 log type classification criteria, as shown in FIG. 5. It is parsed/classified (in Figure 5, it is assumed that it is accurately classified into 5 equal parts) and log data analysis is performed by applying the rule set set for each log type, so 10,000 operations are performed.

As described above, delays and errors frequently occur in analysis because the conventional method cannot cover the rapidly increasing amount of data. However, these problems can be solved through the data analysis and processing device 100 according to the embodiment of the present invention. It becomes possible.

In addition, in Figure 5, it is assumed that log types #1 to #5 all have 100 rules in the ruleset, but in reality, they are classified in detail and contain a smaller number of rules, which has the effect of further reducing actual operations. .

Referring to FIG. 7, the data analysis and processing device 100 according to an embodiment of the present invention includes a first search module capable of searching past log data.

The processor 110 may search log data from a corresponding database based on the target equipment and target period for which past log data analysis is required, and control the analysis unit 160 to perform analysis on the retrieved log data.

In one embodiment, if the log data corresponds to threat information as a result of S300's determination, the processor 110 generates an alarm signal and stores the alarm signal generation details in the database.

If the number of alarm signals generated during a preset time exceeds the threshold range, the processor 110 generates an inspection request signal for the ruleset in which threat information is detected.

In detail, if a risk situation is judged to be higher than the standard for a specific rule included in the rule set, it means determining that there may be an error in the rule and requesting a control agent/manager to inspect the rule.

In one embodiment, the processor 110 sets log type classification criteria so that log data received from security equipment is uniformly distributed across a plurality of nodes, based on the amount of data classified and accumulated in a plurality of nodes over a preset period of time. can be set.

In detail, when the proportion of log data received from at least one first security device among a plurality of different security devices occupies a preset level or more within the proportion of the total log data, the processor 110 determines the first security device. The log type classification criteria for equipment can be set to be more detailed than the preset log type classification criteria.

At this time, subdividing the log type classification criteria means dividing the log type classification criteria into a plurality of sub-criteria, that is, dividing the log type classification criteria and nodes for the relevant security device into multiple nodes.

① The number of nodes can be expanded to subdivide the log type classification criteria.

For example, assume that IDS: 20,000 logs, IPS: 20,000 logs, WAF: 20,000 logs, FW: 120,000 logs, and WEB: 20,000 log data were collected for 1 hour.

In this case, the amount of log data collected from FireWall is too concentrated, and analysis of log data only in that node may be delayed.

Accordingly, the processor 110 subdivides the FW into 1st FW to 4th FW, which expands the log type classification criteria for FW to 4, and the total log type classification criteria (number of nodes) increases to 8, Expected future log data collection is IDS: 20,000, IPS: 20,000, WAF: 20,000, WEB: 20,000, 1st FW: 30,000, 2nd FW: 30,000, 3rd FW: 30,000, 4th FW: The number is expected to be 30,000, which has the effect of appropriately distributing log data.

② The security equipment classification for each node can be rearranged to subdivide the log type classification criteria.

For example, assume that IDS: 20,000 logs, IPS: 20,000 logs, WAF: 20,000 logs, FW: 120,000 logs, and WEB: 20,000 log data were collected for 1 hour.

In this case, the amount of log data collected from FireWall is too concentrated, and analysis of log data only in that node may be delayed.

Dividing the number of nodes by the total number of log data results in 40,000 log data for each of the 5 nodes.

The processor 110 relocates the IDS and IPS to the first node, WAF, and WEB to the second node, the first FW to the third node, the second FW to the fourth node, and the third FW to the fifth node. You can.

Once relocation is completed in this way, the expected log data collection in the future is expected to be 40,000 for each of the first to fifth nodes, which has the effect of appropriately distributing log data.

The method according to an embodiment of the present invention described above may be implemented as a program (or application) and stored in a medium in order to be executed in conjunction with a server, which is hardware.

The above-mentioned program is C, C++, JAVA, machine language, etc. that can be read by the processor (CPU) of the computer through the device interface of the computer in order for the computer to read the program and execute the methods implemented in the program. It may include code coded in a computer language. These codes may include functional codes related to functions that define the necessary functions for executing the methods, and include control codes related to execution procedures necessary for the computer's processor to execute the functions according to predetermined procedures. can do. In addition, these codes may further include memory reference-related codes that indicate at which location (address address) in the computer's internal or external memory additional information or media required for the computer's processor to execute the above functions should be referenced. there is. In addition, if the computer's processor needs to communicate with any other remote computer or server in order to execute the above functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes regarding whether communication should be performed and what information or media should be transmitted and received during communication.

The storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers that the computer can access or on various recording media on the user's computer. Additionally, the medium may be distributed to computer systems connected to a network, and computer-readable code may be stored in a distributed manner.

The steps of the method or algorithm described in connection with embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

Above, embodiments of the present invention have been described with reference to the attached drawings, but those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

100: data analysis processing device
110: processor
120: Department of Communications
130: storage unit
140: input/output unit
150: parsing unit
160: analysis department
170: Search unit

Claims (10)

A communication unit that communicates with a plurality of different security devices; and
When a large amount of log data larger than a preset size is received in real time from the plurality of different security devices through the communication unit, the received log data is parsed according to a preset log type classification standard, and the parsed log data is classified into each log data. Threat information is determined by analysis based on a rule set set in the log type classification criteria, and the received log data is uniformly distributed to the plurality of nodes based on the amount of data classified and accumulated in the plurality of nodes during a preset period. It includes a processor that sets the log type classification criteria to be distributed,
The processor,
When the proportion of log data for a specific log type occupies more than a preset level, the log type classification criteria for the node is expanded to a plurality,
Large-capacity log data analysis device.
According to paragraph 1,
It further includes a cluster consisting of a plurality of nodes,
The plurality of nodes are individually equipped with an analysis application corresponding to the log type set for each node,
The processor performs parallel and distributed processing of log data received from the plurality of different security devices by executing the analysis application provided for each node.
Large-capacity log data analysis device.
According to paragraph 1,
The rule set includes at least one rule for determining whether log data received from the plurality of security devices corresponds to threat information,
A separate analysis module is allocated depending on the type of log data,
The processor analyzes the log data using a rule set set according to each log type,
Large-capacity log data analysis device.
delete According to paragraph 1,
The classification criteria for each log type are determined based on the plurality of different types of security equipment,
The processor is characterized in that when the proportion of log data for a specific log type occupies a preset level or more, the processor rearranges the security equipment classification for each node so that the received log data is distributed.
Large-capacity log data analysis device.
According to paragraph 1,
The data analysis processing device further includes at least one database classified according to a period for which the parsed log data is stored,
The processor stores the parsed log data in the at least one database and deletes the log data after a preset storage period from the point of storage.
Large-capacity log data analysis device.
According to clause 6,
It further includes an analysis module that analyzes past log data,
The processor generates past data analysis results by analyzing the corresponding database based on the target equipment and target period for which past log data analysis is required.
Large-capacity log data analysis device.
According to paragraph 1,
The processor,
As a result of the determination, if the log data corresponds to threat information, an alarm signal is generated,
Characterized in generating an inspection request signal for the ruleset in which threat information is detected when the number of alarm signals generated during a preset time exceeds the threshold range,
Large-capacity log data analysis device.
A method performed by a data analysis processing device,
Receiving real-time log data from a plurality of different security devices;
parsing the received log data according to preset log type classification criteria;
Analyzing the parsed log data based on a rule set set for each log type classification standard to determine whether it is threat information; and
Comprising the step of setting the log type classification criteria so that the received log data is uniformly distributed to the plurality of nodes based on the amount of data classified and accumulated in the plurality of nodes for a preset period of time,
The data analysis processing device,
When the proportion of log data for a specific log type occupies more than a preset level, the log type classification criteria for the node is expanded to a plurality,
Method for analyzing large volume log data.
A computer-readable recording medium that is combined with a hardware computer and stores a large-capacity log data analysis program for executing the large-capacity log data analysis method of claim 9.