KR20100068352A - B-tree index vector based web-log restoration method for huge web log mining and web attack detection - Google Patents

Abstract

PURPOSE: A weblog recovering method based on b-tree index vector for large capacity web log mining and attack detection is provided to perform field unit parsing about web log information and efficient indexing process, thereby improving attack detection and a real-time dealing performance about a large-capacity web log based on mining algorithm. CONSTITUTION: A field division module(20) divides a web log of the log load module(10) by a field. A B-tree indexing module(30) constitutes B-tree through the divided fields. The B-tree indexing module generates an index list and an indexing log. A query processor(40) searches B-tree corresponding to a search condition about external query. The query processor extracts a field index set. A log restoration module(60) restores the web log through an extracted indexing log list.

Description

B-Tree Index Vector Based Web-Log Restoration Method For Huge Web Log Mining And Web Attack Detection}

The present invention relates to a non-index vector-based web log recovery method for large-scale web log mining and attack detection, and more specifically, to construct a non-tree by comparing keywords through a non-tree index technique, and to generate web log information. The present invention relates to a non-index vector-based web log recovery method for large-scale web log mining and attack detection that enables efficient search and recovery of web logs by realizing an efficient indexing structure of information generated therein.

The domestic Internet usage rate is steadily increasing, and the number of domestic Internet users is over 34,430 thousand at present. In June 2007, the Internet usage rate for the age of 6 and over (% of Internet users within the last month) reached 75.5%. As such, the number of domestic Internet users and usage rate is steadily increasing, and the age group and occupation group of users are also diversifying.

As the number of internet users increases, the amount of web logs generated at major portal web sites in Korea is about 50GB / day. As the number of web services increases, the number of vulnerabilities in web services is increasing rapidly. As a result, web attack attempts and success rates on major domestic portal sites are also increasing.

In addition, with the advent of Web 2.0, Web services are expanding and web portal sites using vulnerabilities such as advertisement (ActiveX), phishing, malware distribution, privacy related attacks, phishing, service hacking, XSS attack, SQL Injection, and script attack Attacks are skyrocketing. In particular, vulnerabilities such as attempting malicious attacks on portal sites or illegally leaking information about web users through attacks such as SQL Injection / Parameter Injection and DoS on web services are being discovered.

Therefore, in order to proactively respond to this type of web attack, it is necessary to first apply a method of detecting attack attempts by analyzing web log information generated from a web site. Web log information is used as a tool for analyzing consumer tendencies and providing more advanced services to companies that use the Internet as a major business and marketing tool. However, the collected web log data is not immediately available for analysis. In order to analyze the user's disposition through web logs and extract statistical data of web sites and at the same time to prevent web attacks in advance, web log information is generated from portal site by applying Web Log Preprocessing. There is a need for a fast indexing technique for large web log information.

However, the existing web IDS system and web mining-based attack detection technique has a problem that the detection rate is lowered when the web IDS / IPS system is run because the web server performs the attack detection process without the fast indexing process for the large web logs.

Existing IDS system detects an attack using rule data targeting IP packet to detect an attack. However, in order to detect attack of web server, it is necessary to detect illegal access from outside or provide anomaly detection function by analyzing web log data generated from web server.

Existing web IDS system uses web attack rule information to detect attack from outside or inappropriate query transmission and abnormal access information in web system based on web log.

However, the existing web IDS system does not efficiently cope with web attacks that are performed in real time because it compares with web attack detection rules without separate preprocessing for a large number of generated web logs.

In addition, the existing web IDS compares with the rules through sequential search without the preprocessing of log files. Therefore, in the worst case, the search results are presented only after a certain time has elapsed after generating the web log.

Therefore, when applying web log mining technique together with web log preprocessing process, it can efficiently detect / respond to attack activities in web server such as malicious code based attack and DoS attack through BotNet. It is expected to be able.

Web mining techniques can be largely divided into content mining, usage information mining, and structure mining techniques. With the recent rise of Web2.0 technology, dynamic web site construction is rapidly increasing. Therefore, the amount of web log information to be analyzed is also increasing rapidly. In order to proactively respond to web-based attacks, indexing and mining techniques should be combined. Real-time analysis can be provided through high-speed indexing of large web log information, and various web attacks can be judged.

FIG. 1 is a conceptual diagram illustrating a process of applying a mining technology to a large amount of web information. The conceptual diagram illustrates a process of detecting an attack by analyzing a log file inside a system and comparing it with a defined rule.

2, in the related art, a high speed processing technique for large-scale web log information is insufficient, and an efficient detection function for attack information is insufficient due to a lack of processing function for string information recorded in log information. In addition, the prior art is mostly at the stage for extracting the usage information of the web user. The prior art uses a mining technique as a process for predicting user's behavior information only from web log information. The prior art is used to extract information about which page a user moves to in the future.

However, if we combine the web log mining technology with the attack detection technology in the web log, the detection rate can be improved. Therefore, it is necessary to combine high-speed indexing technology for large web logs.

As described above, web mining refers to automatically discovering web page user access patterns through web log analysis. In general, a web log is generated in a web server that forms a certain format. Web logs generated by IIS or Apache servers follow the W3C format.

3 illustrates a W3C extended log format of IIS 6.0 used in the related art. Of the various log fields of FIG. 3, the field used for web usage mining is used by extracting only information of a specific field in consideration of a complex annual relationship of some fields. Therefore, in order to derive an error detection method and extract mining information from a large amount of web log information, the correlation of attack detection information for each log field as shown in FIG. 4 was derived.

As shown in FIG. 4, a log field element extraction process should be performed in order to detect web abnormal behavior using a large amount of web log information and to connect with a mining technique by applying attack detection. In addition, the pre-processing module of the web log should be developed in advance, such as the process of refining and filtering irrelevant log information, classifying users, classifying the classified users' sessions, and extracting the moving paths from the web site considering the classified sessions.

The web log preprocessing method of the prior art can be largely divided into five steps as shown in FIG.

-Step 1 (Cleaning Log)-This is a preprocessing process that deletes log information that is not necessary for analysis. In general, multimedia files constituting web pages such as jpg, jpge, gif, swf are automatically requested when a user requests a web page, and are unnecessary information for web mining. Therefore, it is deleted. The deleted log can see efficiencies of 1/20 and 1/40.

Step 2 (User Identification)-Identifying the user using web log information. In general, users are identified using the c-ip and c (User Agent) field information. However, when using a proxy server, there is a problem that it is not possible to identify different users using the same computer by simply c-ip and user agent information.

-Session Identification-A session does not indicate whether or not a user is logged in. The session ID is assigned when the user uses the web page in the browser. Therefore, it is the most important preprocessing process for analyzing the movement path and preference in each user's web page.

-Step 4-Path Completion-Tracks the user's movement path in the web page using the classified session information, and generally performs the path connection process when the user moves by pressing the back or forward button.

Step 5-Formatting-Converts the format into information suitable for web analytics. The format type depends on the analysis technique used.

The above web log is used as a useful material for web mining techniques. However, due to the development and change of the modern web environment, the data form of the web log is becoming more complicated and the size is also larger, and the existing preprocessing method is urgently required to be improved.

Due to the development of the web, the size of the web log is increasing. Microsoft.com averages 200-300GB of web logs per day, with up to 1TB of web logs. It is inefficient to preprocess large web logs like this. In order to perform the first step (Cleaning Log) of the existing preprocessing step, it is possible to check the suffix of the log information. For example, all log information with the file name suffix 'gif, jpeg, etc. can be deleted. However, in the existing pretreatment technique, the cleaning log process is performed through the sequential inspection of the entire field in FIG. 3.

In order to generate new information by using field information of each web log, a classification method for each field and a search function within a result for a specific string log are required. For example, log information requesting a specific resource to a web server at regular intervals can be reported as a web scan attack. This result can be detected by extracting the cs-uri-stem field and searching within the results of the date and time fields. However, the existing web log preprocessing technique has an inefficient problem in mining using complex log fields due to the lack of field-based searching and search in results.

An object of the present invention is to use a B-Tree Index Vector structure to retrieve and recover a large amount of web log information at high speed. To provide a recovery method.

According to a preferred embodiment of the present invention for achieving the above object,

In the method of recovering a web log having a bit-index vector-based structure of a predetermined format for attack detection of a web ID system,

A first step of obtaining corresponding log index information from the list;

Incorporating a string by using an index in a non-tree of a corresponding field;

A third step of recovering log text is provided, including a non-index vector-based web log recovery method for mass web log mining and attack detection.

As described above, according to the non-index vector-based weblog recovery method according to the present invention, by extracting the cs-uri-stem field to index the duplicated log information to perform a cleaning log process is more efficient than the conventional.

According to the present invention, it is possible to perform field-by-field parsing and efficient indexing of web log information, and it is expected to improve attack detection and real-time response performance for large logs based on a mining algorithm. IPS system can be built.

According to the present invention, the log information can be loaded, and the size of the log information can be innovatively reduced by using the field-based indexing technique considering field-specific parsing techniques and duplicate strings. It can be configured to improve the performance of the existing pretreatment process.

In addition, according to the present invention, since the multithread-based fast preprocessing and retrieval function for a large amount of web log information is provided, when combined with an abnormal detection and mining technique in the web log information, the overall system performance can be improved.

1 is a conceptual diagram illustrating an example of a conventional large-scale web log preprocessing method.
2 is a conceptual diagram illustrating an example of a conventional web log information based mining method.
Figure 3 shows the W3C extended log format of the prior art (IIS 6.0).
FIG. 4 illustrates abnormal behavior considerations for each field of FIG. 3.
5 is a conceptual diagram illustrating an example of a conventional web log preprocessing technique.
6 is a schematic conceptual diagram of a web log preprocessing method according to the present invention.
7 is a conceptual diagram illustrating a multi-thread based web log indexing process in the web log preprocessing method according to the present invention.
8 is a flowchart illustrating an example of a non-tree indexing vector-based fast weblog search method according to the present invention.
9 shows a non-tree structure in the non-tree indexing vector-based weblog fast search method according to the present invention.
10 is a conceptual diagram illustrating a web log recovery method in the non-trigger indexing vector-based web log fast retrieval method according to the present invention.
FIG. 11 is a diagram illustrating an example of a fast search method in a non-tree indexing vector-based web log fast search method according to the present invention.
12 is a schematic block diagram of a non-tree indexing log processor in the non-tree indexing vector-based weblog fast search method according to the present invention.
13 is a flowchart of FIG. 12.

Hereinafter, the non-index vector-based weblog recovery method according to the present invention will be described in detail with reference to the accompanying drawings.

6 is a schematic conceptual diagram of a web log preprocessing method according to the present invention, FIG. 7 is a conceptual diagram illustrating a multithreaded web log indexing process in a web log preprocessing method according to the present invention, and FIG. FIG. 9 is a flowchart illustrating an example of a non-tree indexing vector-based weblog fast search method according to the present invention. FIG. 9 illustrates a non-tree structure in the non-tree indexing vector-based weblog search method according to the present invention. 10 is a conceptual diagram illustrating a weblog recovery method in the non-trigger indexing vector-based weblog fast search method according to the present invention, and FIG. 11 is a non-trigger indexing vector-based weblog fast search method according to the present invention. 12 is a diagram illustrating an example of a non-tree indexing log processor in a non-tree indexing vector-based weblog fast searching method according to the present invention. Of a block diagram, Figure 13 is a flow chart of Fig.

Referring to FIG. 6, the non-index vector-based web log fast searching method according to the present invention performs a fast preprocessing / indexing process for web log information in order to increase the efficiency of attack detection in a large web log. In addition, we propose a method that can be linked with the log indexing-based mining technique to provide real-time attack response technology. The detailed structure of each module is as follows.

The IIS log is a huge piece of textual information. But one giant piece of information can be broken down into lines, and each line can be broken down into fields. This is because each line consists of fields, and each field has its own information. Therefore, rather than retrieving the necessary information directly from the large sized text information, the partition conquest method that solves the problem is applied, and the multi-threaded fast indexing technique is provided (see FIG. 7).

The proposed method is to index the information of each field and create a B-tree optimized for search. To analyze the web log, you need to convert it to a B-tree at the time the web log is loaded. Therefore, the above operation is executed at the same time as loading the web log.

The processing of duplicate strings is executed automatically when the web log load function is executed. The map module of the STL (Standard Template Library) consists of keywords and data, and consists of B-trees by comparing the keywords.

The large web log information includes time information, sender IP and receiver IP information, and the user's connection environment and access URI information. Therefore, in case of a large portal site, an efficient data structure should be used to efficiently store / manage web log information generated by several tens of GB per day and apply it to attack detection and web mining.

Therefore, the present invention implements an efficient indexing structure for the information generated in the log information by applying the B-tree structure.

Keywords are designated as field information, and data is designated as index of field information. If the information of 'Key5' enters the map, if there is already indexing information in the B-tree, the log information can be stored / managed using the index value that has already been assigned without performing the additional indexing process.

In the present invention, the B-tree indexing vector-based weblog information preprocessing method as shown in FIG. 8 is performed. Converting large web log information to B-tree-based index vector values will reduce the overall storage capacity. Also, in the search process in the web log information, if the search process is performed by the index value rather than the search for the string, the result may be faster. Therefore, in the present invention, B-tree-based index value can be given to IIS web log information to improve performance. As a result, the web log information is stored after the index-based preprocessing process as shown below, minimizing the overall preprocessing time and file size.

The web log has index key value list information and a B-tree structure as shown in FIG. The index key value list stores unique key value information for each web log field, and also provides a fast recovery to the web log original text.

Since each web log information is indexed and stored in a condensed form as shown in FIG. 10, when recovering to the web log original text value, look at the corresponding log index vector value and find the index key value stored in the B-tree. Combining the values, you can reconstruct the original Web log information.

Since each field has index-based B-tree information, it is possible to minimize the time delay required for reconstruction by searching the stored information for each field and reconfiguring it in a multithreaded manner.

With this structure, performance can be maximized even when searching a large amount of web log information.

As shown in FIG. 11, when the index number of a field is known using a web log index vector value, a structure for directly accessing field information using the index number is provided.

When you start a search, you basically search with the information in the field. In that case, we look for information in the optimized B-tree of the search and find the index value for it. After that, all work is done with only the index number obtained. After obtaining the index number, you can refer to the value only by the index number without having to search the B-tree again.

Referring to FIG. 12, the non-tree-based indexing log processor 100 divides a log load module 10 for loading an input web log and a field for dividing the web logs transmitted from the log load module 10 by fields. A non-tree indexing module 30 for constructing a non-tree, creating an index list, and generating an indexing log, by forming a partitioning module 20 and fields divided by the field partitioning module 20, and an external A query processor 40 which searches a non-tree meeting a search condition through a non-tree for a query, extracts a set of field indexes, and requests a modified query; and an indexing log according to the request of the query processor. And a log recovery module 60 for recovering and outputting a log through the indexing log extracting unit 50 and the indexing log list extracted.

Referring to FIG. 13, when a log is loaded first (S1), a log line is extracted (S2), and whether there is field information thereafter (S3). If there is no field information, field information is generated (S4). If there is field information in step S3, field division is performed (S5), and B-tree indexing is performed (S6). After that, it is determined whether it is a new keyword (S7), and if it is a new keyword, a keyword is inserted, an index is generated (S8), and an index is obtained (S9). When the index is generated, the keyword is found in the B-tree field (S24), and the found index is transferred (S9). If it is not a new keyword, an index is obtained (S9), and after the step S9, an indexing log is generated (S10). The generated indexing log is stored in the indexing log field (S25).

Meanwhile, the query processor analyzes the input query at the time of query input (S21), extracts the indexing log in association with the B-tree and the indexing log field (S23), and recovers the log (S26).

10: log load module
20: field division module
30: non-tree indexing module
40: query processor
50: indexing log extractor
60: log recovery module
100: non-tree based indexing log processor

Claims (1)

In the method of recovering a web log having a bit-index vector-based structure of a predetermined format for attack detection of a web ID system,
A first step of obtaining corresponding log index information from the list;
Incorporating a string by using an index in a non-tree of a corresponding field;
The third step of recovering the log text; Bit re-index vector based web log recovery method for mass web log mining and attack detection, characterized in that made.

Images