KR20230106083A - Device, method and program that analyzes large log data using a distributed method for each log type - Google Patents

Abstract

The present invention relates to a data analysis and processing apparatus for each log type for a large-capacity event log, which parses large-capacity log data received from security equipment according to log type classification criteria, and parses the parsed log data according to log type classification criteria. By analyzing based on the rule set, there is an effect of performing parallel distributed processing on log data received from security equipment.

Description

Device, method and program that analyzes large log data using a distributed method for each log type}

The present invention relates to an apparatus for analyzing log data, and more particularly, to an apparatus for analyzing large-volume log data using a distributed method for each log type.

In SIEM (Security Information Event Management) or CEP (Complex Event), Event Log Data generated by various equipment is analyzed in real time to determine whether there is a threat from inside or outside.

However, it is possible to install additional SIEM/CEP modules on all operating server equipment and allocate resources separately within each server to process all log data from reception to analysis. There are many cases where smooth operation is impossible due to shortage or the like.

In addition, when the analysis process is performed within each server, the analyzed result is very fragmentary, and it is also difficult to obtain continuous results of the entire equipment by connecting with the analyzed results from other servers.

Therefore, if a separate server for SIEM / CEP can be prepared to collect all event logs into one and process them in batches, overall and continuous analysis results can be obtained from log data gathered from many devices.

In the prior art, various types of log data received from various security devices were collectively analyzed. However, as the amount of data has increased rapidly recently, it is impossible to process all log data with this analysis method.

Accordingly, a technology for analyzing and processing large-capacity log data in a method different from the prior art is required, but such a technology has not been disclosed at present.

Publication No. 10-2018-0061891, (2018.06.08)

The present invention to solve the above problems is to parse large-capacity log data received from security equipment according to log type classification criteria, and analyze the parsed log data based on a rule set set for each log type classification criteria to detect threats. We want to determine whether information is available or not.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

An apparatus for analyzing and processing data for each log type for an event log according to an embodiment of the present invention for solving the above problems includes a communication unit that communicates with a plurality of different security devices; and when log data of a predetermined size or larger is received in real time from the plurality of different security devices through the communication unit, the received log data is parsed according to a predetermined log type classification criterion, and the parsed log data is Threat information is determined by analysis based on a rule set set for each log type classification criterion, and the received log data is sent to the plurality of nodes based on the amount of data accumulated and classified in the plurality of nodes during a predetermined period. and a processor configured to set the log type classification criterion to be uniformly distributed.

Further, a cluster consisting of a plurality of nodes is further included, and each of the plurality of nodes is individually provided with an analysis application corresponding to a log type set for each node, and the processor is provided for each node. By executing the analysis application, log data received from the plurality of different security devices may be processed in parallel and distributed.

In addition, the rule set includes at least one rule for determining whether the log data received from the plurality of security devices corresponds to threat information, and separate analysis modules are allocated according to the type of log data. The processor may analyze the log data using a rule set according to each log type.

In addition, when the proportion of log data for a specific log type is greater than or equal to a predetermined level, the processor may expand the log type classification criterion for the corresponding node to a plurality.

In addition, the log type classification criterion is determined based on the plurality of different types of security equipment, and the processor determines that the received log data is stored when the proportion of log data for a specific log type is greater than or equal to a predetermined level. Security equipment classification by node can be rearranged to be distributed.

The data analysis and processing device further includes at least one database classified according to a period in which the parsed log data is stored, and the processor stores the parsed log data in the at least one database, and stores the parsed log data at a storage time point. The log data may be deleted after a preset storage period.

The processor may further include an analysis module for analyzing past log data, and the processor may analyze a corresponding database based on a target device and a target period for which past log data analysis is required to generate a past data analysis result.

In addition, the processor generates an alarm signal when the log data corresponds to threat information as a result of the determination, and checks a rule set in which threat information is detected when the number of alarm signals generated for a predetermined time exceeds a threshold range A request signal can be generated.

In addition, a data analysis and processing method for each log type for an event log according to an embodiment of the present invention for solving the above problems is a method performed by a data analysis processing device, and a real-time log log from a plurality of different security devices. receiving data; parsing the received log data according to a preset log type classification criterion; determining threat information by analyzing the parsed log data based on a rule set set for each log type classification criterion; and setting the log type classification criterion so that the received log data is uniformly distributed to the plurality of nodes based on the amount of data classified and accumulated in the plurality of nodes for a predetermined period of time.

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium recording a computer program for executing the method may be further provided.

According to the present invention as described above, large-capacity log data received from security equipment is parsed according to log type classification criteria, and the parsed log data is analyzed based on a rule set set for each log type classification criteria, thereby receiving data from security equipment. There is an effect of performing parallel distributed processing for the log data to be processed.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a block diagram of an apparatus for analyzing and processing data for each log type for an event log according to an embodiment of the present invention.
2 is a flowchart of a data analysis and processing method for each log type for an event log according to an embodiment of the present invention.
3 is a diagram illustrating the configuration of a cluster in an embodiment of the present invention.
4 is a diagram illustrating a conventional method of analyzing and processing data for each log type for an event log.
5 is a diagram illustrating a data analysis and processing method for each log type for an event log according to an embodiment of the present invention.
6 is a diagram illustrating a configuration diagram of a data analysis processing apparatus for each log type for an event log according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an apparatus for analyzing and processing data for each log type for an event log according to an embodiment of the present invention.

2 is a flowchart of a data analysis and processing method for each log type for an event log according to an embodiment of the present invention.

3 is a diagram illustrating the configuration of a cluster in an embodiment of the present invention.

4 is a diagram illustrating a conventional method of analyzing and processing data for each log type for an event log.

5 is a diagram illustrating a data analysis and processing method for each log type for an event log according to an embodiment of the present invention.

6 is a diagram illustrating a configuration diagram of a data analysis processing apparatus for each log type for an event log according to an embodiment of the present invention.

A cluster (or computer cluster) is a concept that connects several computers through a network so that they can be used as a single computer. Computer clusters are used in various fields such as computer operating systems, computer hardware, and statistical data. The components of a computer cluster are usually connected by a high-speed local area network.

A cluster generally boasts greater performance and reliability than a single computer, and is much more efficient than a single computer.

Referring to FIG. 1 , an apparatus 100 for analyzing and processing data for each log type for an event log according to an embodiment of the present invention includes a processor 110, a communication unit 120, a storage unit 130, an input/output unit 140, It includes a parsing unit 150, an analysis unit 160, a search unit 170 and a log receiving module.

However, in some embodiments, the data analysis processing apparatus 100 may include fewer or more components than those shown in FIG. 1 .

The communication unit 120 communicates with a plurality of different security devices, field manager terminals, etc., specifically, receives real-time log data from a plurality of different security devices, and the processor 110 generates a threat information discovery signal for the log data. Then, the signal can be transmitted to the site manager terminal.

The communication unit 120 may further include a log receiving module 190 responsible for only receiving logs.

The storage unit 130 includes at least one storage means, and for example, log data parsed and classified by the parsing unit 150 is stored.

In addition, the storage unit 130 may store various commands and algorithms for executing data analysis and processing methods, and various artificial intelligence models may be stored together.

The storage unit 130 may further include a log type storage unit for storing log types.

The input/output unit 140 may be composed of separate input means and output means, and may receive various control signals from the manager and field manager through the input means, and may receive threat information or not through the output means, an error You can print messages, etc.

The parsing unit 150 parses the log data received through the log receiving module according to preset log type classification criteria.

The parsing unit 150 may include at least one parsing module, and may include separate parsing modules according to log types, so that each parsing module may parse log data according to log type classification criteria.

In the embodiment of the present invention, the parsing unit 150 is illustrated as parsing the received log data according to the log type classification criterion, but the log type classification criterion is only a top priority in the parsing of the parsing unit 150, and parsing/parsing/ The classification criterion is not limited to one log type classification criterion.

For example, classification criteria such as SQL query where, group by having, etc. may be further applied.

The analysis unit 160 may include at least one analysis module, including separate analysis modules according to log types, and each analysis module is parsed from the parsing unit 150 and classified for each log type. Analysis can be performed on the log data.

In this case, each analysis module may include an analysis application and analyze log data using a rule set optimized for each log type.

The search unit 170 may execute a search function for log data stored in the storage unit 130, and includes at least one search module, specifically, a first search module for integrated analysis of past data, and a real-time search function. It may include a second search module and the like for.

As an embodiment, a search module may be included in the analysis unit 160 for real-time search.

The processor 110 is in charge of controlling the components in the data analysis processing device 100, and executes/uses commands, algorithms, artificial intelligence models, etc. stored in the storage unit 130 to process data analysis according to an embodiment of the present invention. method can be executed.

Referring to FIG. 2 , a process of a data analysis processing method according to an embodiment of the present invention will be described in detail.

The processor 110 receives real-time log data from a plurality of different security devices through the communication unit 120 . (S100)

Specifically, a plurality of different security devices received through the communication unit 120 are analyzed and processed, and as shown in FIGS. 4 and 5, various security devices operated by clients such as IDS, IPS, WAF, FW, and WEB are applied. It is possible, and other security equipment can be tied to ETC.

The processor 110 parses the log data received in S100 according to a preset log type classification criterion. (S200)

As an embodiment, as described above, the parsing unit 150 may include separate parsing modules according to log types, and log data received through the communication unit 120 are analyzed by each parsing module as a log type classification criterion. can be parsed according to

And, the processor 110 stores the log data parsed and classified through the parsing unit 150 in the storage unit 130 according to the classification result.

In detail, the processor 110 may store the log data in the storage unit 130 as classified through the parsing unit 150 to facilitate real-time analysis, search for past data in the future, and analysis.

At this time, the storage unit 130 includes at least one database classified according to the storage period of the log data classified through the parsing unit 150 .

For example, the storage unit 130 may include a first database for storing log data for 1 week, a second database for storing log data for 1 month, and a third database for storing log data for 1 year. .

The processor 110 stores the parsed log data in at least one database and deletes the log data after a preset storage period from the storage time.

The processor 110 determines threat information by analyzing the log data parsed in S200 based on a rule set set for each log type classification criterion. (S300)

In an embodiment of the present invention, the rule set stores at least one rule for determining whether log data corresponds to threat information, and such a rule can be set by a security control officer or a person in charge.

In addition, the processor 110 may generate a warning to the input/output unit 140, the security control device, and the terminal when the specific log data corresponds to the threat information through S300 so that the security control personnel/person in charge can confirm it.

The data analysis processing apparatus 100 according to an embodiment of the present invention includes a cluster composed of a plurality of nodes.

Each node is individually provided with an analysis application corresponding to a log type set for each node.

The processor 110 may perform parallel distributed processing of a large amount of log data received from a plurality of different security devices by executing an analysis application provided for each node.

In an embodiment of the present invention, classification criteria for each log type may be determined based on a plurality of different types of security equipment.

4 is a diagram illustrating a conventional method of analyzing and processing data for each log type for an event log.

In the embodiment of the present invention, there are 5 types of security devices operated by clients, and accordingly, 5 types of log types are illustrated.

Referring to FIG. 4, conventionally, when 100 pieces of log data are received, 5 types of log types and 100 rules are calculated for all of the 100 pieces of log data when the analysis process is performed, resulting in 100 x 5 x 100 = 50,000 operations performed.

However, when the data analysis and treatment device according to the embodiment of the present invention to which the above configurations are applied is applied, 100 log data received from the security equipment are divided into 5 log types according to the 5 log type classification criteria as shown in FIG. It is parsed/classified (assuming that it is accurately classified into 5 equal parts in FIG. 5), and log data analysis is performed by applying the ruleset set for each log type, so 10,000 operations are performed.

As described above, the conventional method cannot cover the rapidly increasing amount of data, so that analysis delays and errors often occur. be able to

In addition, in FIG. 5, it is assumed that all of the log types #1 to #5 have 100 rules in the rule set, but in reality, since they are classified in detail and contain fewer rules, the actual operation is further reduced. .

Referring to FIG. 7 , the data analysis and processing apparatus 100 according to an embodiment of the present invention includes a first search module capable of searching log data in the past.

The processor 110 may search log data from a corresponding database based on target equipment and target period for which analysis of past log data is required, and control the analyzer 160 to analyze the searched log data.

As an embodiment, the processor 110 generates an alarm signal and stores the alarm signal generation history in a database when the log data corresponds to threat information as a result of the determination in S300.

The processor 110 generates a check request signal for a rule set in which threat information is detected when the number of alert signals generated during a preset time exceeds a threshold range.

In detail, when a dangerous situation is judged to be greater than the reference value for a specific rule included in the rule set, it means that it is determined that there may be an error in the rule, and a control agent/manager is requested to check the rule.

In an embodiment, the processor 110 may perform log type classification criteria such that log data received from the security device is uniformly distributed among the plurality of nodes based on the amount of data that has been classified and accumulated in the plurality of nodes for a predetermined period of time. can be set.

In detail, the processor 110 may perform the first security device when the ratio of log data received from at least one first security device among a plurality of different security devices occupies a predetermined level or more within the weight of the total log data. Log type classification criteria for equipment may be set to be more subdivided than preset log type classification criteria.

At this time, subdividing the log type classification criterion means dividing the log type classification criterion into a plurality of sub-criteria, that is, dividing the log type classification criterion and node for the security device into several.

① The number of nodes can be expanded for segmentation based on log type classification.

For example, assume that IDS: 20,000, IPS: 20,000, WAF: 20,000, FW: 120,000, and WEB: 20,000 log data are collected for one hour.

In this case, the amount of log data collected from FireWall is too concentrated, and log data analysis can be delayed only from that node.

Therefore, the processor 110 subdivides the FWs into first FWs to fourth FWs, whereby the log type classification criteria for the FWs are expanded to 4, and the total log type classification criteria (number of nodes) increases to 8, Log data collection expected in the future is IDS: 20,000, IPS: 20,000, WAF: 20,000, WEB: 20,000, 1st FW: 30,000, 2nd FW: 30,000, 3rd FW: 30,000, 4th FW: 30,000 is expected to have the effect of properly distributing log data.

② Security equipment classification by node can be rearranged to subdivide the log type classification criteria.

For example, assume that IDS: 20,000, IPS: 20,000, WAF: 20,000, FW: 120,000, and WEB: 20,000 log data are collected for one hour.

In this case, the amount of log data collected from FireWall is too concentrated, and log data analysis can be delayed only from that node.

Dividing the number of nodes by the total number of log data is calculated as 40,000 log data for each of the five nodes.

The processor 110 relocates the IDS and IPS to the first node, the WAF and WEB to the second node, the first FW to the third node, the second FW to the fourth node, and the third FW to the fifth node. can

When the relocation is completed in this way, log data collection expected in the future is expected to be 40,000 from the first node to the fifth node, respectively, and the log data is appropriately distributed.

The method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a server, which is hardware, and stored in a medium.

The aforementioned program is C, C++, JAVA, machine language, etc. It may include a code coded in a computer language of. These codes may include functional codes related to functions defining necessary functions for executing the methods, and include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include memory reference related codes for which location (address address) of the computer's internal or external memory should be referenced for additional information or media required for the computer's processor to execute the functions. there is. In addition, when the processor of the computer needs to communicate with any other remote computer or server in order to execute the functions, the code uses the computer's communication module to determine how to communicate with any other remote computer or server. It may further include communication-related codes for whether to communicate, what kind of information or media to transmit/receive during communication, and the like.

The storage medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and is readable by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers accessible by the computer or various recording media on the user's computer. In addition, the medium may be distributed to computer systems connected through a network, and computer readable codes may be stored in a distributed manner.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: data analysis processing unit
110: processor
120: communication department
130: storage unit
140: input/output unit
150: parsing unit
160: analysis unit
170: search unit

Claims (10)

a communication unit that communicates with a plurality of different security devices; and
When a large amount of log data of a predetermined size or larger is received in real time from the plurality of different security devices through the communication unit, the received log data is parsed according to a predetermined log type classification criterion, and the parsed log data is respectively Threat information is determined by analysis based on a rule set set in the log type classification criteria, and the received log data is uniformly distributed to the plurality of nodes based on the amount of accumulated data classified in the plurality of nodes during a predetermined period. A processor for setting the log type classification criterion to be distributed in a distributed manner;
Large volume log data analysis device.
According to claim 1,
Further comprising; a cluster consisting of a plurality of nodes,
The plurality of nodes are individually provided with an analysis application corresponding to a log type set for each node,
Characterized in that the processor performs parallel distributed processing of the log data received from the plurality of different security devices by executing the analysis application provided for each node.
Large volume log data analysis device.
According to claim 1,
The rule set includes at least one rule for determining whether the log data received from the plurality of security devices corresponds to threat information,
Each separate analysis module is allocated according to the type of log data,
The processor analyzes the log data using a rule set set according to each log type.
Large volume log data analysis device.
According to claim 1,
the processor,
Characterized in that, when the proportion of log data for a specific log type occupies more than a predetermined level, the log type classification criteria for the corresponding node are expanded to a plurality.
Large volume log data analysis device.
According to claim 1,
The log type classification criterion is determined based on the plurality of different types of security equipment,
Characterized in that the processor rearranges the classification of security equipment for each node so that the received log data is distributed when the proportion of log data for a specific log type occupies a predetermined level or more.
Large volume log data analysis device.
According to claim 1,
The data analysis and processing device further includes at least one database classified according to a period in which the parsed log data is stored,
The processor stores the parsed log data in the at least one database, and deletes the log data after a preset storage period from a storage time.
Large volume log data analysis device.
According to claim 6,
Further comprising an analysis module for analyzing past log data,
Characterized in that the processor generates past data analysis results by analyzing the corresponding database based on the target equipment and target period for which past log data analysis is required,
Large volume log data analysis device.
According to claim 1,
the processor,
As a result of the determination, when the log data corresponds to threat information, an alarm signal is generated;
Characterized in that, when the number of alert signals generated during a preset time exceeds a threshold range, a check request signal for a rule set in which threat information is detected is generated.
Large volume log data analysis device.
A method performed by a data analysis processing device,
Receiving real-time log data from a plurality of different security devices;
parsing the received log data according to a preset log type classification criterion;
determining threat information by analyzing the parsed log data based on a rule set set for each log type classification criterion; and
Setting the log type classification criterion so that the received log data is uniformly distributed to the plurality of nodes based on the amount of data classified and accumulated in the plurality of nodes for a predetermined period of time.
Methods for analyzing large volumes of log data.
A computer-readable recording medium in which a large-capacity log data analysis program for executing the large-capacity log data analysis method of claim 9 is stored in combination with a computer, which is hardware.

Images