KR101435789B1 - System and Method for Big Data Processing of DLP System - Google Patents

Abstract

The present invention discloses a data processing system and method. A data processing server according to an aspect of the present invention includes: a receiving unit for receiving log data from at least one DLP (Data Loss Prevention) server; A classifier for classifying the fields of the log data by predetermined index object fields; An index generator for generating index data of the log data using the classified index target field and storing the generated index data together with the index ID of the log data in a storage in one index node; And storing the unstructured data among the log data in a plurality of HDFSs in an HDFS (Hadoop Distributed File System) cluster, associating the formatted data in the log data with the index ID and the location information including the irregular data, And a storage management unit for distributively storing the data in the plurality of hives HIVE.

Description

Technical Field [0001] The present invention relates to a system and method for processing large data in a DLP system,

The present invention relates to a distributed processing system, and more particularly, to a data processing system and method capable of distributed processing high capacity data.

Wikipedia defines Big Data as a large set of formal or unstructured data sets that go beyond the ability to collect, store, manage and analyze data from existing database management tools.

Big data processing technology means collecting, storing, managing and analyzing a large amount of fixed or unstructured data sets to extract values or analyze desired results.

Big data processing technology, on the other hand, is defined as architecture technology that extracts value from a wide variety of large scale data at low cost, and is capable of ultra-fast acquisition, discovery and analysis.

In recent years, big data and big data processing technologies have been used in various fields of society by a representative open source platform called Hadoop and its extended elements. Specifically, big data processing technology is increasingly utilized in political and social fields, economic fields, management fields, and cultural fields, and contributes to creating new values and markets for human society.

In the field of information protection, also known as DLP (Data Loss Prevention), there is an attempt to incorporate a big data processing technology.

The amount of logs produced per day in enterprise and institutional DLP systems has increased from tens to hundreds of gigabytes. This increases log management costs, and in terms of compliance, the human / temporal cost of retrieving the desired logs from logs accumulated over months or years has become very high.

For example, if a log of a DLP system produces a log of 20 gigabytes per day, the log of one year is about 5 terabytes, and if a log of 100 gigabytes of data per day is produced, the log of one year is as much as 24 terabytes . Therefore, in order to retrieve a desired log from among them, a time from a minimum of several hours to several days and a human cost are required.

As a practical example, W Bank searched for approximately 10 terabytes of DLP logs accumulated over three years to submit evidence of financial accidents by its employees, using three persons and log search and database equipment It took months of time.

As a result, the DLP system stores logs in a relational database, maintains scalability for increasingly large logs, and distributes / stores logs appropriately using multiple log databases. Of course, a relational database is extremely efficient for processing structured or structured data and can store a large amount of data. However, storing and managing larger amounts of data requires high-cost investments.

Furthermore, since the logs generated by the DLP system are mixed with the fixed data and the irregular data, it is difficult to quickly search the log data, and additional human, temporal, and physical costs are required.

For example, the type of formatted data in the DLP system's log is the type of protocol, the transmission / reception time, the sending / receiving IP address, the sender / receiver ID or e-mail address, the presence or absence of the attachment, Personal information, financial information, business strategy, investment strategy, product planning information, service planning information, development source / drawing, etc.). In addition, types of unstructured data include email title, email body, instant messaging message, blog title, blog body, and text content and file source of attached or uploaded files.

FIG. 1 is a block diagram illustrating a conventional DLP server and a DB server for storing and searching large-capacity logs.

As shown in FIG. 1, a plurality of DLP servers collectively distribute a large amount of DLP logs and simultaneously distribute and store the DLP logs in a plurality of DBs. Accordingly, in order to retrieve a desired log among the large-capacity logs, That is, if a total of three years worth of logs is stored in three DB servers for one year, it is necessary to perform a process of retrieving a log of one year from each DB three times in order to retrieve a desired DLP log. If the DLP log is searched by a search condition made up of a combination of fixed and unstructured data, it takes several tens of hours or several days depending on the complexity.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a data processing system and method capable of distributing DLP logs in a distributed manner.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

A data processing server according to an aspect of the present invention includes: a receiving unit for receiving log data from at least one DLP (Data Loss Prevention) server; A classifier for classifying the fields of the log data by predetermined index object fields; An index generator for generating index data of the log data using the classified index target field and storing the generated index data together with the index ID of the log data in a storage in one index node; And storing the unstructured data among the log data in a plurality of HDFSs in an HDFS (Hadoop Distributed File System) cluster, associating the formatted data in the log data with the index ID and the location information including the irregular data, And a storage management unit for distributively storing the data in the plurality of hives HIVE.

An index ID of log data in an HDFS (Hadoop Distributed File System) cluster according to another aspect of the present invention, a plurality of hives (HIVE) for distributing and storing fixed data including meta data, A data processing server for retrieving log data according to a request from a plurality of HDFSs may share a search request including a data pair corresponding to a search condition and a keyword entered by a user to another management unit of another data processing apparatus, The index data corresponding to the data pair is retrieved from the repository in the index node, the retrieval result of the index data corresponding to the data pair in the other index node of the other management unit is confirmed, A job manager for generating a corresponding index ID list; And an output unit for querying information corresponding to the index ID list with the plurality of hives and receiving and outputting information corresponding to the index ID list from at least one of the plurality of hives in response thereto .

According to the present invention, it is possible to reduce the time and cost of the distributed processing of the large-capacity DLP log data at the level of the big data.

FIG. 1 is a block diagram illustrating a DLP server and a DB server for storing and retrieving large-capacity logs in the related art.
2 is a configuration diagram showing a data processing system according to an embodiment of the present invention;
3 is a detailed configuration diagram of a data processing system according to an embodiment of the present invention;
4 illustrates a portion of an index function calling code according to an embodiment of the present invention.
5A and 5B illustrate a process of automatic balancing between index nodes according to an embodiment of the present invention.
FIG. 6 illustrates a data pair for an index search request in JSON format according to an embodiment of the present invention; FIG.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. As used herein, the terms " comprises, " and / or "comprising" refer to the presence or absence of one or more other components, steps, operations, and / Or additions.

A specific embodiment of the present invention will now be described with reference to the accompanying drawings.

FIG. 2 is a configuration diagram illustrating a data processing system according to an embodiment of the present invention, and FIG. 3 is a detailed configuration diagram of a data processing system according to an embodiment of the present invention. FIG. 4 is a diagram illustrating a portion of an index function calling code according to an embodiment of the present invention. FIGS. 5A and 5B illustrate a process of automatic balancing between index nodes according to an embodiment of the present invention. FIG. 6 is a diagram illustrating a data pair for an index search request in a JSON format according to an embodiment of the present invention.

2 and 3, the data processing system 30 according to the embodiment of the present invention includes an index cluster 300, an HDFS cluster 400, and a search server 200.

The index cluster 300 includes at least one of the index servers 300_1 to 300_n and is used for indexing and searching operations of the DLP log data.

Each of the index servers 300_1 to 300_n includes a receiving unit 340, a classifying unit 330, an index generating unit 320, a storage managing unit 310, an index node 380, a cluster managing unit 350, 370 and a search management unit 360.

The search server 200 is a middleware system for high-speed search, searches for DLP logs according to a user's request, and can be configured based on the web. The search server 200 includes a search request unit 210, a request conversion unit 240, a result reception unit 220, and a result output unit 230.

The HDFS cluster 400 is a Hadoop-based distributed data store and includes at least one of the HDFS servers 400_1 to 400_n. Each of the HDFS servers 400_1 to 400_n includes a HIVE (Big Data Warehouse System for Hadoop) 410 and a HDFS (Hadoop Distributed File System)

Here, the hive 410 is a data warehouse system used in Hadoop, that is, a database system specialized for processing big data, and is a repository in which formal metadata of DLP log data is stored. The HDFS 420 is a distributed file system used in Hadoop, and is a repository in which unstructured data among DLP log data is stored in file units.

Hereinafter, each part of the data processing system 30 in the DLP log data distribution process and the DLP log data retrieval process of the data processing system 30 will be described.

① DLP  Distributing Log Data

Each DLP server 100 sequentially transmits DLP log data by a round robin method by the log transfer unit 110. [ The distributed storage of DLP log data is mainly performed by one of the index servers 300_1 to 300_n. Hereinafter, each part of the index servers 300_1 to 300_n in the index cluster 300 will be described with reference to FIG. do.

The receiving unit 340 receives the DLP log data transmitted from at least one DLP server 100 and transmits the DLP log data to the classifying unit 330. At this time, the DLP log data is transmitted from the log transfer unit 110 of the DLP server 100.

Upon receiving the DLP log data from the receiver 340, the classifying unit 330 classifies the detailed fields of the DLP log data and requests indexing of the detailed fields of the classified DLP log data as shown in FIG. 4 .

Specifically, the classifying unit 330 classifies the DLP log data into index target fields by checking preset index target fields in each DLP log data. The classifying unit 330 extracts the text stream from the unstructured data including the file included in the DLP log data, classifies the text stream into one of the index target fields, and transmits the text stream to the index generating unit 320. Here, the classifying unit 330 may divide the index object field of one DLP log data from the index object field of another DLP log data, and transmit the index object field to the index generating unit 320.

Here, the indexing target field is predefined by the administrator, and the type of the protocol, the transmission time, the reception time, the transmission IP address, the reception IP address, the sender ID, the receiver ID, the e-mail address, An email title, an email body, an instant messaging message, a blog title, a blog body, and a text stream and file name extracted from an attached or uploaded file. Here, the sensitive information may be information requiring security such as personal information, financial information, business strategy, investment strategy, product planning information, service planning information, development source, drawing, and the like. Then, the type of sensitive information and the number of detections for each identification code are detected by the index server and included in the DLP log data.

The index generator 320 indexes index target fields of the classified DLP log data to generate index data (i.e., index ID and index data), and stores the index data in the storage allocated to the index node 380. [

For example, the index generator 320 assigns a different index ID to each DLP log data, and generates at least one key text of each DLP log data as index data. Specifically, the index generator 320 performs morphological analysis on the text stream of the irregular data field excluding the format data or the structured data field in the index object field of the DLP log data. Then, full text indexing is performed on the text according to the morphological analysis result to generate at least one index data which is a core test of each DLP log data. Here, the structured data or structured data field may be a data field having an IP address, a time, and a numeric value of integer type. For example, the index generating unit 320 may analyze the morphemes of the English, Korean, and numbers of the unstructured data fields in the N-Gram scheme and may generate a text stream of " Morphological analysis can be performed in the form of "entering" and "going".

The index generator 320 sequentially stores the generated index data in the main storage in the index node 380 selected in the round robin manner among the main stores in the index node 380.

The index node 380 is provided for each of the index servers 300_1 to 300_n, and has as many primary and replicate repositories as the number designated by the administrator, and stores the index data. In this case, the number of index nodes is preferably set to three or more for efficient automatic balancing and load balancing between index nodes. Here, the main repository stores the index data, and the replica repository replicates the main repository, and is used as a backup for failover.

The cluster management unit 350 performs automatic balancing and load balancing between index nodes through automatic relocation of the main storage and replica storage between the index nodes in the index cluster 300. [ Of course, the capacity of the index data is smaller than that of the DLP log data, but as the index data increases, the capacity of the index data increases, which may lead to a decrease in the speed of DLP log search. In addition, if any one of the index nodes fails, a problem may arise in distributed storage and distributed retrieval of the index nodes. Therefore, the cluster management unit (i) performs load balancing between the index nodes.

To this end, the cluster management unit 350 periodically exchanges state information with the cluster management units of all the index servers existing in the index cluster 300. Then, Cross Health Checking is performed to check the fault status of index nodes in other index servers using the exchanged state information. Therefore, the cluster management unit 350 according to the present invention can ensure high availability and flexible scalability between the index operation and the index data.

For example, when the main storage is 3, the replica storage for each major storage is designated as 1, and the index node 380 is gradually expanded by the cluster management unit 350 to 3 in total, the cluster management unit 350 The automatic relocation procedure may be as shown in FIG. 5A. Specifically, the cluster management unit 350 manages not to put a replica of one major storage and its corresponding major storage in one index node together by using index nodes of other index servers.

As another example, if a failure occurs in the index node 3 while all three index nodes are used as shown in FIG. 5A, the cluster manager of the index server to which the index node 1 and the index node 2 belongs automatically detects the failure. Therefore, as shown in FIG. 5B, the cluster manager of the index server to which the index node 2 having the replica 3 as the replica of the primary 3 is included, creates the main store 3 using the replica store 3 . In addition, the cluster manager of the index server to which the index node 3 belongs newly creates the replica store 1 as the replica store for the main store 1.

As described above, according to the present invention, when failure occurs in any one of the index nodes during operation of three index nodes, automatic negotiation between the cluster management units can be automatically performed to operate as two index nodes, thereby ensuring high availability of the index node can do.

When the indexing operation for each DLP log data is completed, the index generating unit 320 adds an index ID according to the indexing operation to each DLP log data, and transmits the index ID to the storage management unit 310. [

The storage management unit 310 distributes the DLP log data to which the index ID is added to the hive 410 and the HDFS 420 in accordance with the interface protocol for processing the Hadoop based data.

Specifically, the storage management unit 310 stores meta data fields of the DLP log data in the form of a table record in the hive 410, and stores unstructured data fields of the DLP log data in the HDFS 420 in the form of a file data set . Here, the metadata can be designated by the administrator, and the DLP log data includes information such as the type of protocol, transmission time, reception time, transmission IP, reception IP, sender ID, receiver ID, e-mail address, Information type, email title, blog title, attachment file name, and the like. The unstructured data can also be an email body, a chat message of a messenger, a blog body, a text stream extracted from a file, a file source, and the like.

At this time, the storage management unit 310 stores the unstructured data in the HDFS 420 in the form of a file data set, checks the location information of each file data set, and hashes the location information of the file data set. The storage management unit 310 stores the location information hash value of the file data set in the form of a table record in the hive 410 of the HDFS servers 400_1 to 400_n together with the index ID and the metadata. In this manner, the storage management unit 310 can store the location information hash value of the file data set in association with the index ID and the metadata.

Meanwhile, those skilled in the art know that the hive 410 and the HDFS 420 constitute a Hadoop-based platform as well as a distributed data processing and service and a high availability guarantee structure, so that a detailed description thereof will be omitted. Therefore, according to the present invention, desired search results can be provided within a few hours even when DLP log data not indexed is retrieved using the Hadoop-based distributed storage and distributed parallel structure.

As described above, according to the present invention, the DLP log data is applied to the HDFS (Hadoop Distributed File System) based on the characteristics of the large data processing platform and technology represented by Hadoop and MapReduce, It can be divided into a set of units, and can be distributedly stored after indexing processing.

In the above, the indexing process for DLP log data and the Hadoop platform-based distributed storing process have been described. Hereinafter, a fast searching method of indexed and distributed DLP log data will be described.

② DLP  Log data retrieval process

The DLP log data retrieval process is mainly performed by the search server 200 and the index servers 300_1 to 300_n. Hereinafter, a process of searching DLP log data by each component of the search server 200 and the index servers 300_1 to 300_n will be described.

Upon receiving the search request by the administrator, the request conversion unit 240 of the search server identifies and analyzes the search condition and the search keyword from the search request, and searches for the search condition and the keyword data pair . At this time, the administrator can access the search server 200 on the web using the manager PC, input the search condition and the search keyword, and search for the corresponding index.

The search request unit 210 of the search server converts the data pair of the "condition item: keyword" type into the JSON format. The search request unit 210 selects an index server in a round robin manner among the index servers in the index cluster 300 and transmits a request message including a data pair of JSON format to the search management unit 360 of the selected index server To request an index search. The data pair of the JSON format for the index search request of the search request unit 210 according to the embodiment of the present invention is shown in FIG.

When the index search request is received, the search management unit 360 of the index server transmits and shares a request message including a data pair of JSON format to another search management unit of another index server in the index cluster 300. Accordingly, the search management unit 360 of all the index servers in the index cluster 300 can perform a search operation on the index node 380 almost at the same time.

At this time, the search management unit 360 and the other search management unit use indexes (i.e., index data) corresponding to the data pair of JSON format in the main storage of the index node 380 provided by each search engine 370, Or not. Then, the search management unit 360 confirms the search results of the index data corresponding to the data pair of the JSON format retrieved by the other search management unit. Then, the search management unit 360 combines search results (index data and index IDs) of the other search management unit with search results (index data and index IDs) directly searched to generate an index ID list matching the search conditions. For example, each search management unit 360 determines whether index data (index data) corresponding to a data pair of JSON format exists from the main storage of the index node 380, .

Then, each search management unit 360 transmits a search result, that is, an index ID list matching the search conditions, to the search server 200.

Then, the result receiving unit 220 receives the received index ID list and transfers the index ID list to the result output unit 230.

The result output unit 230 outputs a table record corresponding to the index ID list (that is, the index ID, the metadata, and the position of the file dataset) in the hive 410 of the HDFS servers 400_1 through 400_n, Information). At this time, the query from the result output unit 230 is transmitted to the hive 410 of all the HDFS servers 400_1 to 400 - n in the HDFS cluster 400.

Upon receipt of the query from the result output unit 230, the hive 410 of each HDFS server checks the response corresponding to the query (i.e., the record set having each index ID in the index ID list) ).

The result output unit 230 outputs the response received from the hive 410 of each HDFS server, that is, the index ID for the DLP log data conforming to the search condition, the file data set of the unstructured data corresponding to the metadata, And the location information hash value - in sequence. For example, the result output unit 230 may output the received responses to the screen in the order in which they are received.

When the administrator confirms the metadata corresponding to the search condition and requests at least one of the output DLP log data to inquire the detail information, the result output unit 230 outputs a file location hash value corresponding to the selected DLP log data And extracts the details of the selected DLP log from the HDFS 420 and outputs it to the screen.

Meanwhile, although the DLP log data retrieval process is performed by the collaboration between the search server and the index server as an example, the DLP log data retrieval process may be performed by one of the search server and the index server Of course.

As described above, the search server 200 according to the embodiment of the present invention does not need to know which DLP server has the index data of the DLP log data corresponding to the search request of the administrator, The search condition is converted into the JSON format and transmitted to one of the index servers 300_1 to 300_n so that a desired search result can be obtained from the index servers 300_1 to 300_n.

In addition, according to the present invention, it is possible to retrieve and inquire desired log data within several seconds to several hours from large DLP log data having a large data level of several terabytes or more accumulated for several months to several years, The human and the temporal costs required for the present invention can be drastically reduced.

In addition, in the present invention, instead of the conventional commercial DBMS and expensive server and storage, a low-cost HDFS cluster composed of a plurality of PC-class servers and DAS (represented by HDD) Storage of log data and high-speed retrieval may be possible. Accordingly, the present invention efficiently collects and stores large DLP log data at a level of 20 to 50% of that of a conventional commercial database while providing higher scalability and availability through distributed parallel search in connection with a search engine. , Manage and analyze.

In addition, in the present invention, at the time of inquiry and output of the primary search result, irregular data such as a file having a relatively large data size as compared with the metadata is excluded and the search result is secondarily provided according to the request of the user, Can be efficiently managed, and the page output performance can be guaranteed. Therefore, in the present invention, it is possible to guarantee the fast search performance of the DLP log data and the index based on the Hadoop-based distributed parallel data processing.

While the present invention has been described in detail with reference to the accompanying drawings, it is to be understood that the invention is not limited to the above-described embodiments. Those skilled in the art will appreciate that various modifications, Of course, this is possible. For example, in the above description, the case where the index server and the search server distribute the DLP log data generated by the DLP server has been described as an example, but the index server and the search server may not be able to perform the log including the unstructured data It goes without saying that the data may be distributed.

Accordingly, the scope of protection of the present invention should not be limited to the above-described embodiments, but should be determined by the description of the following claims.

Claims (12)

A receiving unit for receiving log data from at least one Data Loss Prevention (DLP) server;
A classifier for classifying the fields of the log data by predetermined index object fields;
An index generator for generating index data of the log data using the classified index target field and storing the generated index data together with the index ID of the log data in a storage in one index node; And
Wherein the non-standard data is distributedly stored in a plurality of HDFSs in a HDFS (Hadoop Distributed File System) cluster, and the format data is associated with the index ID and the location information including the irregular data in the HDFS cluster A storage management unit for distributing and storing the data in a plurality of hives (HIVE)
And the data processing server. The method according to claim 1,
Sharing state information of another index processing node in another data processing server and another index processing node in the other data processing server, and using the status information to relocate the one index node and the other index node through the storage relocation between the one index node and the other index node A cluster manager for balancing load balancing of index nodes
The data processing server further comprising: The information processing apparatus according to claim 1,
Wherein the location information hash value obtained by hashing the location information including the irregular data is stored in the form of a table record in association with the index ID and the form data. The apparatus according to claim 1,
Extracts a text stream from a file included in the log data, and classifies the text stream into one of index target fields including the atypical data. 2. The apparatus of claim 1,
And morphologically analyzing the atypical data of the log data to extract key text therefrom to generate the index data for the log data. The information processing apparatus according to claim 1,
And distributes the unstructured data in the form of a file data set. 2. The data processing apparatus according to claim 1,
The type of protocol of the log data, the transmission time, the reception time, the transmission IP address, the reception IP address, the sender ID, the receiver ID, the e-mail address, the presence of the attachment file, the e-mail title, the blog title, The type of the sensitive information included in the log data, and the number of detections by the identification code of the sensitive information type. 2. The method according to claim 1,
An email body, an instant messaging message of a messenger, a blog body, and a text stream extracted from an attachment or an upload file. An index ID of log data in an HDFS (Hadoop Distributed File System) cluster, a plurality of hives (HIVE) for distributedly storing the formatted data including the metadata, and a plurality of HDFSs distributedly storing the unstructured data in the log data, A data processing server for retrieving log data,
A search request including a data pair corresponding to a search condition and a keyword entered by a user is shared with other management units of the other data processing apparatuses and index data corresponding to the data pair is retrieved from a storage in the provided index node, A job manager for confirming a search result of the index data corresponding to the data pair in the other index node of the other management unit and generating an index ID list corresponding to the data pair by combining the search results; And
An information processing unit for querying information corresponding to the index ID list with the plurality of hives and receiving and outputting information corresponding to the index ID list in at least one of the plurality of hives,
And the data processing server. 10. The method of claim 9,
The index ID of the log data corresponding to the data pair, the metadata, and the location information hash value of the file data set. The image processing apparatus according to claim 10,
When the user requests detailed information inquiry about the information corresponding to the index ID list, the file data set is fetched from the HDFS corresponding to the location information hash value of the plurality of HDFSs using the location information hash value, The data processing server. 12. The method of claim 11,
A conversion unit for identifying the search condition and the keyword in a search request input by the user and converting the search condition and the keyword into data pairs of a "condition item: keyword" type in JSON format,
The data processing server further comprising:

Images