KR101543377B1 - Apparatus and method for analyzing data using mapreduce based on nosql - Google Patents

Abstract

A data analysis apparatus using MapReduce based on NoSQL (Structure Only Query Language) is a NoSQL based database configured to store input data having one or more fields and values corresponding to each of the one or more fields; A map module configured to extract the contents of a predetermined field from the input data to generate parsed data and generate first output data obtained by filtering the parsed data based on a predetermined rule; And a Reduce module configured to receive the first output data from the map module and generate the second output data by merging the first output data according to the contents of the predetermined field.

Description

[0001] APPARATUS AND METHOD FOR ANALYZING DATA USING MAPREDUCE BASED ON NOSQL [0002]

Embodiments relate to an apparatus and method for analyzing data using MapReduce based on NoSQL (Not-Only Structured Query Language (SQL)).

Recently, as the scope of the service industry using networks has expanded, such as the spread of computers and the Internet, the spread of smart devices including smart phones, and the increase of credit cards and online commerce, many companies are interested in information protection system It is going out. Organizations are struggling with operational management and the implementation of consistent information protection policies as the information protection system grows. Integrated security control systems have played an effective role in those areas where early analysis of intelligently changing information security breaches is needed.

In order to efficiently operate the enterprise system, the security control system quickly grasps the source of the failure by monitoring the system state, the incoming traffic and the data, and various kinds of analysis rules (Rule) , It is possible to judge their hazard, to respond to security threats, and to build an alarm system. In addition to corporations, we are building a security control system to solve the problems of timely action, accident prevention and follow-up in real-time through comprehensive analysis and monitoring in the overall operation and judgment of national computer networks and IT convergence systems in various organizational forms .

As such, with the development of ICT (Information & Communication Technology) technology and convergence with various industries, demand for security equipment is increasing. As a result, the amount of data that is generated in various devices is also increasing, and it is expanding to the size of big data, which is huge data that can not be covered by the existing management system.

Big data is sometimes referred to as "Very Large data", "Extreme data", or "Total data". In the beginning, the amount of data was used as a standard, . However, nowadays, with the development of IT technology, various kinds of data generated in various IT devices are also diversified, which means a large amount of structured or untructured data sets that are difficult to collect / store / analyze / do.

The key issue of this big data technology is that it complements the performance limitations of the existing database and provides the platform with a platform for storage, analysis and systematization, and the information necessary to efficiently use the vast amount of data collected at the right time Data cleansing techniques that can be collected are described as representative. In the integrated security control system, big data technology is introduced to handle the big data class security logs that flow in real time more efficiently, thereby lowering the temporal processing cost and error ratio, We need technology that can guarantee rate.

However, there are many difficulties in processing and managing data in such existing systems. In the conventional systems using a relational database (for example, an RDBMS-based database), the scale-out of the system itself is expanded in processing large data. The storage space is insufficient and the speed reduction problem occurs, which may hinder smooth service. As a solution to this problem, a method of reducing the amount of data that has been increased by purchasing and installing a separate data storage and processing solution or storing the data in a binary form within the system is proposed. However, the data currently used in most IT convergence systems is not as costly as it is not as important as using expensive data management solutions. In addition, it is not suitable for systems that need to analyze and process in real time because it requires a separate file conversion process when converting to binary form.

Therefore, there is a need for a new paradigm capable of storing a large amount of data at a low cost and quickly and efficiently refining the collected data to construct an optimized data set. In other words, it can analyze and monitor the status of many system equipments in real time, cope with emergency and emergency situations, and prevent mass accidents such as logs, data, events, , Analysis technology suitable for big data environment is needed which enables construction of comprehensive response system and alarm system capable of judging harmfulness and real time response.

Patent Publication No. 10-2012-0078908

According to an aspect of the present invention, there is provided a database based on NoSQL (Structured Query Language (NoSQL)) capable of processing a large number of security logs more efficiently, and a system for removing unnecessary information It is possible to provide an integrated security control technology capable of coping with the Big Data era by the preprocessing algorithm technique using the MapReduce design which can extract information only and use it in a timely manner.

A data analysis apparatus according to an embodiment includes: a NoSQL (Structured Query Language (NoSQL) -based database configured to store input data having one or more fields and contents corresponding to each of the one or more fields; A map module configured to extract the contents of a predetermined field from the input data to generate parsed data and generate first output data obtained by filtering the parsed data based on a predetermined rule; And a Reduce module configured to receive the first output data from the map module and generate the second output data by merging the first output data according to the contents of the predetermined field.

According to an embodiment, a method of analyzing data includes storing input data in a NoSQL-based database having one or more fields and contents corresponding to each of the one or more fields; Extracting contents of a predetermined field from the input data to generate parsed data; Filtering the parsed data based on the content of the predetermined field to generate first output data; Merging the first output data based on the contents of the predetermined field to generate second output data; And outputting the second output data.

A computer-readable storage medium, according to one embodiment, may store instructions for performing the method of data analysis by a computer, executed by a computer.

According to the apparatus and method for analyzing data according to an aspect of the present invention, it is possible to construct a big data storage engine capable of storing raw data itself in a database in a large volume log collection and analysis system. Also, by extracting and selecting essential features through preprocessing of big data in IT convergence system, it is possible to improve computation and processing efficiency and apply it to big data flowing in real time. The above technology is IT fusion? The system can provide a basis for real-time searching, indexing, and / or correlation analysis of data, events, or traffic. In addition, it is possible to detect a specific attack from the data through the refined data composed of the optimized features, improve the accuracy in data retrieval and indexing processing, and reduce the error rate.

1 is a schematic block diagram of a data analysis apparatus according to one embodiment.
2 is a flowchart of a data analysis method according to an embodiment.
FIG. 3 is a schematic diagram for explaining a data collection and refinement process according to a data analysis method according to an embodiment.
FIG. 4 is a graph illustrating a data collection rate according to a data analysis method according to an exemplary embodiment in comparison with a conventional technique.
FIG. 5 is a graph illustrating a data retrieval speed according to a data analysis method according to an exemplary embodiment in comparison with a conventional technique.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a schematic block diagram of a data analysis apparatus according to one embodiment.

1, the data analysis apparatus includes a database 10 based on a NoSQL (Structured Query Language) database, and a Map module 20 for performing data analysis based on MapReduce. And a Reduce module 30. The terms "database "," module ", or "device" and the like are used herein to refer to a combination of hardware and software driven by that hardware. For example, the hardware may be a data processing device comprising a CPU or other processor. Also, the software driven by the hardware may refer to a running process, an object, an executable, a thread of execution, a program, and the like.

In the database 10, input data for analysis is received. In the present specification, the present invention will be described on the basis of an embodiment in which a firewall log is analyzed using a data analysis apparatus. In the present embodiment, the database 10 includes a firewall, a web firewall, a VPN (Virtual Private Network), an Intrusion Detection System (IDS), an Intrusion Prevention System (IPS) ), SMS (System Management Solution), or other types of security devices. However, this is illustrative, and in other embodiments the data analysis device may be utilized for other types of Big Data analysis and processing other than firewall log analysis.

In embodiments, the database 10 may be a NoSQL-based database, such as HBase, Cassandra, MongoDB, and the like. The manner of processing data based on conventional relational databases (e.g., MySQL or RDBMS) does not reflect the current nature of rapidly changing and gigantic security log data and has limitations in analysis. The relational database needs to set up a complex relationship between each log data table and analyze it. Therefore, it is difficult to reflect the changing pattern of the visitor or the new attack type. That is, you must re-establish the relationship each time in order to modify or add a changed passenger pattern or a new attack.

Therefore, in the embodiments of the present invention, the NoSQL-based database 10 is used instead of the relational database. As a result of using the NoSQL-based database 10, the map module 20 and the redistribution module 30 can store various types of data in the database 10 using a key value without a fixed data schema It is accessible. Thus, by modifying the rules for detecting attacks, new attack types can be quickly reflected.

The map module 20 and the reduction module 30 are parts for analyzing data by performing MapReduce-based data processing on a part or all of the input data stored in the database 10. When the data analysis apparatus according to the embodiments is used for analyzing the security log data, the map module 20 parses the log data stored in the database 10 and performs an attack from the parsed data according to a predetermined rule It can detect. In addition, the reduction module 30 may receive the data output from the map module 20 and output the set of data merged according to the reference, thereby constructing the output data.

2 is a flowchart of a data analysis method according to an embodiment. 1 and 2, a data analysis method according to an embodiment will be described below.

First, input data may be collected in the NoSQL-based database 10 (S1). When applying the data analysis method according to the embodiments to the firewall log analysis, the NoSQL-based database 10 can be input from a firewall, a web firewall, a VPN, an IDS / IPS, an NMS / SMS, It is possible to receive log data as data. The input data collected in the database 10 may include one or more fields and content corresponding to each field. The fields of data and the contents of each field can be implemented in various forms according to the application of data analysis. For example, when the input data is log data, the input data may be configured as shown in Table 1 below.

field Content and format Data Time Date and time of log occurrence YYYYMMDD HHMMSS format
(Y: year, M: month, D: day, H: hour, M: minute, S: second) Source IP (Source IP) IP address of log origin 2 * 2 bytes (unsigned String) Destination IP (Detection IP) IP address of log origination destination 2 * 2-byte unsigned string Destination port
(Destination Port) Port of the log origination destination 2 * 2-byte unsigned string Protocol protocol 2 * 2-byte unsigned string IP: 0 / ICMP: 1 / CGP: 3 / TCP: 6
EGP: 8 / PUP: 12 / UDP: 17 Action Code Details of the log message A string of up to 256 bytes 1 (Allowed): Allow / Accept / Close
2 (Deny): Drop / Reject

However, what is described in Table 1 is merely an example, and the form of data collected in the database 10 according to the embodiments is not limited to the above.

Next, the input data collected in the database 10 can be parsed by the map module 20. The map module 20 may be operated using a Map () function that receives a pair of key and value and outputs the parsed data as a new key-value pair using the key. For example, in one embodiment, the pseudo code of the Map () function for driving the map module 20 is shown in Table 2 below.

Map () Function pseudo-code while ( LogList  ! = Null ) {
map :
function () {
this . attack .forEach ( function ( attack _ type ) {
   emit ( attack  , { date , count : One});
}

However, the pseudo-code of the Map () function described above is merely exemplary, and in the data analysis apparatus according to embodiments, the map module 20 may be designed to operate using functions written in other different forms or written in different languages .

The map module 20 uses a predetermined field to be used for attack detection among one or more fields included in the input data as a key of the Map () function and stores the contents of the corresponding field in the map () ) Can be used to perform parsing of input data. The predetermined field to be parsed may be one or a plurality of predetermined fields and may be appropriately determined according to the attack type by the user and input to the map module 20. In one embodiment, the data analysis apparatus may further include an input module (not shown) for the user to input predetermined fields and attack detection rules and the like.

Next, the map module 20 can filter data determined as an attack in the parsed data by applying a predetermined attack detection rule to the parsed data. The attack detection rule serving as a criterion for performing the filtering may be appropriately determined by the user and input to the map module 20, and may be modified or newly generated according to the change of the attack type. Some examples of attack detection rules will be described in detail below with reference to FIG. 4 and FIG. Unlike the prior art, since parsing of data and detection of abnormal data in the map module 20 are processed at once, it is possible to reduce the number of analysis steps, which can contribute to cost reduction when implemented in a device.

Among the data parsed by the map module 20, data determined as an attack may be output from the map module 20 as first output data and transmitted to the reduction module 30. [ The first output data may be a pair of a new key and a value to be used in the reduction module 30. [ For example, the first output data may include contents of a specific field (e.g., Date Time) used for attack detection among the input data and count information (count) of data detected as an attack.

The reduction module 30 may receive the first output data from the map module 20 and may merge the first output data with respect to the key to generate second output data. The reduction module 30 can be operated using a Reduce () function that receives a pair of a key and a value and merges data using the key. For example, in one embodiment, the pseudo code of the Reduce () function for driving the reduction module 30 is shown in Table 3 below.

Reduce () Function pseudo-code while ( LogList  ! = Null ) {
reduce :
   function ( key , values ) {// ( date , count )
total  = 0;
   for (n = 0; n < values . length ; n + +) {
total  + = values [n]. count ;}
return { attack : total };

However, the pseudo-code of the Reduce () function described above is merely exemplary, and in the data analysis apparatus according to the embodiments, the reduce module 30 may be implemented using functions written in different different forms or written in different different languages .

The reduce module 30 receives the content of a specific field (e.g., Date Time) used for attack detection among the first output data output from the map module 20 as a key of the Reduce () function And the aggregation information (count) of the data detected by the attack can be used as the value of the Reduc () function. The merged data includes an aggregation of the log data in which the attack occurred, which may be output from the reduction module 30 as second output data.

In one embodiment, the data analysis device may further comprise a display module (not shown) for displaying the second output data for viewing by the user. The user can confirm the second output data through the display module and perform various analyzes on the firewall logs through the number of attacks or attack types counted at a specific time.

FIG. 3 is a schematic diagram for explaining a data collection and refinement process according to a data analysis method according to an embodiment.

In the example shown in Fig. 3, the input data has fields of Date, Source IP (Src IP), Destination IP (Dst IP), Protocol (for example, ICMP) And a data group (Attacks 1-3). The Map () function performs parsing using the Date field and corresponding contents as keys and values, respectively. As a result of parsing, the input data is parsed including the pair of the contents of the Date field (e.g., 2012706 134, 2012706 164, 2012706 165) and the corresponding aggregation information (e.g., count (1) Lt; / RTI > The Map () function generates the first output data that has been determined to be an attack among the parsed data, and transmits it to the Reduce () function.

The Reduce () function returns the contents (e.g., "2012706 134", "2012706 164", "2012706 165") and corresponding aggregation information (eg, count The data are merged using key and value, respectively. For example, the Reduce () function merges the data having the same log generation date and time, and the Reduce () function generates second output data including the sum of the aggregation information of the data having the same log generation date and time, do. In FIG. 3, it is exemplarily shown that there are five data having the log generation date and time "2012706 164" and two data having the log generation date and time "2012706 165".

FIG. 4 is a graph illustrating a data collection rate according to a data analysis method according to an exemplary embodiment in comparison with a conventional technique.

In the result shown in Fig. 4, the conventional technique used the MySQL-5.6.10 version, which is the most used database in the relational database. As the NoSQL-based database according to the embodiment of the present invention, MongoDB-2.4.1 version was used. The data used in the experiment is 10,000,000 out of the 12,847,649 security log data of the actual company. The rules of attack detection used in the experiment are as follows.

i. Attack 1: More than 5 ICMP messages for 5 minutes from the same source to the same destination

ii. Attack 2: More than 50 ICMP messages in 5 minutes from 50 sources to the same destination

iii. Attack 3: Over 10000 communication occurrences per minute from the same source to the same destination.

The rule is set to judge that the data corresponding to one or more of attack 1 to attack 3 is an attack, and the SQL query performance is tested using the rule, and the attack suspicion scenario is detected and analyzed.

Figure 4 shows the results of an insertion test, which is a test that tests how quickly log data can be collected, or how quickly the collected log data can be retrieved. The graph 410 of FIG. 4 shows the results of the prior art based on the MySQL database, and the graph 420 shows the results of the MongoDB based embodiment of the present invention. As can be seen, as the number of input data increases, the speed difference between the prior art and the embodiment of the present invention increases remarkably.

The results shown in FIG. 4 are summarized in Table 4 below. When 2 million data are inserted, the speed difference between the conventional technique and the present embodiment is 936.275 seconds, and when 10 million data are inserted, 4946.733 seconds, and the speed difference was increased about 5 times. This indicates that the difference between the speed of inserting data as the data is increased proportionally increases.

division 2 million 5 million 10M Examples of the present invention 59.022 152.107 340.533 Conventional technology 995.298 2470.203 4946.733 Speed difference (seconds) 936.275 2318.096 4606.2

FIG. 5 is a graph illustrating a data retrieval speed according to a data analysis method according to an exemplary embodiment in comparison with a conventional technique.

5 shows a result of a search test as to how quickly and accurately the user can collect desired information from the data stored in the database. As in FIG. 4, FIG. 5 also compares the prior art with the embodiment of the present invention while increasing the number of data to 2 million, 5 million and 10 million. The graphs 510 and 520 in FIG. 5 represent results by the MongoDB-based embodiments of the present invention, and the graphs 530 and 540 represent results by the MySQL-based prior art. In addition, the graphs 510 and 530 correspond to a case where the number of search queries is three, and the graphs 530 and 540 correspond to a case where the number of search queries is four.

As shown, embodiments of the present invention exhibit faster performance rates than the prior art, regardless of the number of query conditions or the number of data. The results shown in FIG. 5 are summarized in Table 5 below. In the embodiments of the present invention, the overall average of search queries is 0.0027 seconds, Stability. On the other hand, in the case of the conventional technology based on MySQL, as the number of data increases, the search speed also increases, and the search query speed increases not only with the increase in data but also with the increase in the number of search query conditions, .

division 2 million 5 million 10M Examples of the present invention
(Search query 3) 0.001 0.0022 0.0037 Examples of the present invention
(Search query 4) 0.002 0.0032 0.0041 Conventional technology
(Search query 3) 5.8435 15.9285 37.9725 Conventional technology
(Search query 4) 5.422 22.6375 46.157

The data analysis method according to the embodiments described above can be at least partially implemented in a computer program and recorded in a computer-readable recording medium. The computer-readable recording medium on which the program for implementing the data analysis method according to the embodiments is recorded includes all kinds of recording apparatuses in which data that can be read by the computer is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and a carrier wave (for example, And the like. The computer readable recording medium may also be distributed over a networked computer system so that computer readable code is stored and executed in a distributed manner.

While the invention has been shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. However, it should be understood that such modifications are within the technical scope of the present invention. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

Claims (11)

A NoSQL based database configured to store input data having one or more fields and contents corresponding to each of the one or more fields;
A map module configured to generate parsed data by extracting contents of a predetermined field from the input data, and generate first output data that is filtered based on a predetermined rule; And
A redistribution module configured to receive the first output data from the map module and to combine the first output data with the contents of the predetermined field to generate second output data,
Wherein the database is a MongoDB (DB).
The method according to claim 1,
Wherein the input data is log data,
Wherein the one or more fields comprise at least one of a date and time, a source IP, a destination IP, a destination port, a protocol and an action code.
3. The method of claim 2,
The predetermined rule includes an attack detection rule based on at least one of a date and time, a source IP, a destination IP, a destination port, a protocol, and an action code,
Wherein the first output data includes data determined as an attack by the attack detection rule among the parsed data.
The method according to claim 1,
Wherein the second output data includes a number of data in which the content of the predetermined field is the same in the first output data.
delete Storing, in a NoSQL-based database, input data having one or more fields and values corresponding to each of the one or more fields;
Extracting contents of a predetermined field from the input data to generate parsed data;
The data analysis apparatus comprising: filtering the parsed data based on contents of the predetermined field to generate first output data;
The data analysis apparatus comprising: merging the first output data based on the contents of the predetermined field to generate second output data; And
And the data analysis apparatus outputting the second output data,
Wherein the database is a MongoDB database.
The method according to claim 6,
Wherein the input data is log data,
Wherein the one or more fields comprise at least one of a date and time, a source IP, a destination IP, a destination port, a protocol and an action code.
8. The method of claim 7,
The predetermined rule includes an attack detection rule based on at least one of a date and time, a source IP, a destination IP, a destination port, a protocol, and an action code,
Wherein the first output data includes data determined to be attacked by the attack detection rule among the parsed data.
The method according to claim 6,
Wherein the second output data includes a number of data in which the content of the predetermined field is the same in the first output data.
delete By being executed by a computer,
Storing, in a NoSQL-based database, input data having one or more fields and values corresponding to each of the one or more fields;
Extracting contents of a predetermined field from the input data to generate parsed data;
Filtering the parsed data based on the content of the predetermined field to generate first output data;
Merging the first output data based on the contents of the predetermined field to generate second output data; And
And outputting the second output data, wherein instructions for performing a data analysis method are stored,
Wherein the database is a MongoDB (DB).

Images