KR20110111863A - Web robot detection system and method - Google Patents

Abstract

The present invention relates to a web robot detection system and method, comprising: at least one or more monitored web servers distributed in the Internet; A web log server which collects a web log generated for each HTTP request from at least one or more monitored web servers and delivers a standard web log in a raw format; Collects standard web logs generated for each HTTP request from the web log server and stores them in the log queue, parses the web logs from the first HTTP request, considering the timeout of the HTTP session, and defines from the parsed information. A web robot detection server detecting a web robot or a normal user by extracting the differentiated features and calculating a feature ratio between the current session and all users; And a monitoring system for receiving a detected result indicating whether the web robot is a normal user from the web robot detection server. Therefore, the web robot detection system detects the web robot based on the relative difference between the value of the composite characteristic vector of the monitored session and the entire user. At this time, the composite property vector value of all users is updated by all HTTP requests generated from the web server, and the vector value of the monitored session is updated by HTTP requests generated from the session.

Description

Web robot detection system and method

The present invention relates to a system and method for detecting a web robot, and more particularly, to collect web logs generated for each HTTP request from a web log server linked with many monitored web servers, and to operate the web robot based on the internet and the web ( The present invention relates to a web robot detection system and a method for effectively detecting a web robot, represented by an angular component and a length component of a composite attribute vector (CAV) of a polar coordinate system in consideration of characteristics of a web robot.

Web Robots are responsible for gathering the necessary information from the web documents on the Internet, indexing them, and organizing them in a database.

In addition, web robots are mainly used in search engines. Web robots used by search engines navigate through web documents around the world and search for indexes of relevant information in their databases. Web robots are also called WWW Wanderer, WWW Crawler, and WWW Spider in the sense of moving around the web.

Most search engines on the Internet use Agents, called Web Robots, that access and retrieve information sources.

A web robot is a program that traverses a web server (HTTP Server), follows the web's hypertext structure, and recursively extracts other documents referred to as hyperlinks from the HTML document. It is a program to collect. The web robot keeps track of the URLs it visits and checks whether they have visited them later.

Web robots keep URLs visited by humans instead of repetitive tasks of following each link on the homepage, and the program analyzes the data itself, extracts the URLs in it, and moves the information to other URLs. It will check if you have visited the URL.

The form of web robot is Googlebot for search engine that searches from database of HTML documents collected in hypertext structure by web robot according to search keyword on internet, and MOM spider, EIT Link Verifier. Robot, web site mirroring (HTMLgobble, GetUrl, WebCopy, WebFetcher), web robot for article collection (NHN web robot).

When a web robot visits a website such as http://www.aaa.pe.kr/, it first calls http://www.aaa.pe.kr/robots.txt file. When a web robot finds a robots.txt file formed at the root of a web site, the web robot reads the file and analyzes the contents, and collects only the content that the web robot can collect with information about access rights and accessible paths to the web robot. You can only have one "robots.txt" on a site.

If the robots.txt file reads as follows, the URL (Uniform Resource Locator) will block all robots from accessing all paths.

user-agent: * # Applies to all robots

disallow: / # prohibit indexing all pages

There must be at least one "disallow" field in the robots.txt file.

user-agent: Set the robot to not allow access.

disallow: Set the list not to allow. If you set "disallow" to an empty value, all subpaths are accessible.

The web robot runs a multi-processor document loading function and a new URL extraction function from the loaded documents for simultaneous collection of multiple Uniform Resource Locators (URLs) on the Internet.

For example, the web robot using the Fork-Join technique is composed of a webpage loader, a URL extractor, and a uniqueness checker. In the web robot, the loader and the extractor are driven by a multiprocessor.

The most common technique for detecting existing web robots is detection method using known web robot information such as “robots.txt”, UserAgent information, head request rate, referer field information, and invisible link. Although there is a detection method, such a method uses only a part of known information of various characteristics of the web robot so that only a specific web robot can be detected.

However, the disadvantage is that web robots can easily trick the detection system. In addition, the technique of detecting web robots by applying machine learning techniques to HTTP traffic and behavior patterns requires a relatively large number of HTTP requests for accurate detection, which results in a complicated processing operation and a long processing time.

The present invention has been proposed to solve the problems of the prior art, and an object of the present invention is to analyze HTTP requests in real time while minimizing false positives and false negatives in consideration of various characteristics of a web robot. It detects web robots early with a small number of HTTP requests and detects web robots at angles and lengths based on complex feature vector values of the polar coordinate system, which is calculated by integrating the differentiating characteristics of web robots into one. Web robots have high detection accuracy compared to existing web robot detection techniques that only consider some features, and provide a platform that can be extended to detect new types of web robots in the future by applying features extracted through further characterization studies in the future. It is to provide a detection system.

In addition, another object of the present invention is to provide a web robot detection method particularly suitable for implementing a web robot detection system.

In order to achieve the object of the present invention, a web robot detection system, at least one or more monitored web server connected to the Internet; A web log server which collects a web log generated for each HTTP request from the at least one monitored web server and delivers a standard web log in raw format; By collecting the standard web log of the raw format form generated for each HTTP request from the web log server, taking into account the differentiation characteristic information extracted from the standard web log of the complex attribute vector (CAV: Composite Attribute Vector) A web robot detection server configured to display angles and lengths to provide detected results indicating whether the web robot is a normal user; And a monitoring system for receiving a detected result indicating whether the web robot is a normal user from the web robot detection server.

In addition, to achieve another object of the present invention, a web robot detection method comprising a web log server, a web robot detection server, and a monitoring system interlocked with at least one web server to be monitored, (a) the at least one Collecting a web log generated for each HTTP request from the monitored web server to a web log server; And (b) collecting standard web logs in a raw format that is generated for each one HTTP request from the web log server to the web robot detection server, parsing web logs from the HTTP request, and parsing the web logs from the parsed information to various web robots. Based on the relative difference of the composite property vector value between the current monitored session (HTTP session) and the entire user, we extract the matching differentiation characteristics and express the angle and length in the form of the composite characteristic vector of the polar coordinate system. Providing the detected results indicative to the monitoring system.

In addition, to achieve another object of the present invention, a computer, (a) the function of collecting a web log generated for each HTTP request from the at least one or more monitored web server to a web log server; (b) The web processing server of the web robot detection server receives the web log generated for each HTTP request from the web log server and stores the web log in the log queue, and repeatedly parses the web log from the first HTTP request. And processing the differentiated characteristic information from the parsed information and storing the same in a predetermined characteristic array; (c) The complex characteristic vector calculation unit of the monitored session extracts the characteristic information of the individual monitored session and the average characteristic information of all the users from the newly processed web log generated according to the one HTTP request, and is currently monitored. Calculate a composite property vector (CAV) between the session and all users, normalize and calculate the angular and length components using the relative difference between the current HTTP session and the average of all users from a web log of one HTTP request, The normalized angular component and the length component are multiplied by the weight component by considering the weighted average to calculate the angle and length of the composite characteristic vector (CAV), and the CAV value for one current HTTP request and the previous previous request including the same. Updating the CAV value of the current monitored session through a vector sum of the CAV values of the entire monitored session; (d) The comparison analysis unit displays the polarity based on the relative difference between the composite feature vector values of the monitored session and the entire user based on the CAV value reflecting the result of the last HTTP request from the composite feature vector calculation unit of the monitored session. Determining whether a web robot is defined according to a defined angular region, checking reliability of a result by a length component, and determining whether the web robot is a normal user or providing a detected result to an alerting and reporting unit; And (e) a computer-readable recording medium having recorded thereon a program for realizing a function of determining whether the web robot is a normal user or a normal user from an alerting and reporting unit and providing the detected result to a monitoring system.

According to the present invention as described above, since the existing research attempted to detect using some characteristics of the web robot (Web Robot), the detection target is limited and the web robot detection is possible only in a specific environment. However, the present invention maintains a high accuracy of web robot detection and effectively and effectively detects web robots by representing them as a composite attribute vector (CAV) in consideration of various characteristics of web robots. There is. When the existing web robot detection system determines whether or not the web robot by considering partial characteristic components individually, the probability of false detection of the web robot increases accordingly. However, the web robot detection system according to the present invention minimizes the false detection rate and false positive rate in consideration of various characteristics of the web robot and guarantees a very low false positive error so as not to damage the normal user.

In addition, since the present invention indicates whether or not the web robot as one indicator, it is possible to intuitively understand the activity by the web robot and quickly determine whether there is an abnormality and take action. Web robot detection system has the advantage of early detection because it can determine whether or not the web robot with a relatively small number of HTTP requests (HTTP Request) compared to the existing research. In addition, the present invention provides an extensible platform capable of detecting web robots using various differentiation characteristics obtained from characterization studies of normal users and web robots in the future for detecting new web robots to be generated in the future.

1 is an overall configuration diagram of a system for detecting a web robot according to the present invention.
2 is a diagram illustrating a data processing and preprocessing process in a data processing and preprocessing unit of a web robot detection system as a preliminary step for detecting a web robot.
3 is a diagram illustrating a process of calculating and updating a composite attribute vector (CAV) of a total user and a monitored session by receiving data preprocessed by a data processing and preprocessing unit of a web robot detection system.
4 shows an area for determining a detection result from a complex feature vector.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the present invention.

The present invention distinguishes between web robot users and normal users by using web robots, which are automated system tools based on the web, on the Internet (Client-Server Bytes, Head Request Ratios, Referer Pages). Rate, HTTP Error rate, image resource and dynamic / static web page rate, Inter-Request Gap time average, and whether or not 'robots.txt' file is requested). By integrating the angular and length components of each differentiation characteristic, a vector is generated on the polar coordinate system, and the web robot is detected using the detection region segmentation model and the calculated vector.

The web robot detection system according to the present invention minimizes the false detection rate and false positive rate by considering various characteristics, not individual considerations of partial characteristics of a web robot, and indicates whether the web robot is a single indicator. Intuitive and efficient way to detect web robots is presented. It has the advantage of providing an extensible platform that enables early detection of web robots with minimal resources and reflects the new differentiation found through additional web robot characterization studies.

This protects sensitive information on the web server, makes efficient use of resources, and supports the detection of new or unknown web robots.

The web robot detection method proposed by the present invention proceeds in three steps.

First, it receives the standard web logs generated from the web log server for HTTP requests to at least one monitored web server on the Internet, and selects the characteristics that show the difference between normal users and web robots through characterization analysis. do. Second, the selected features are converted into a single feature vector value. This step again consists of two processes: normalization and integration. First, the selected characteristics are normalized by angle and length components so that they can be applied to composite attribute vectors (CAVs). Each normalized feature value is integrated into a single representative value, that is, a complex feature vector value through a weighted average and a length weight. Third, it is determined whether the web robot is based on the angle region defined on the polar coordinate system based on the calculated CAV, and the reliability of the result is confirmed by the length component. The present invention detects a web robot based on the relative difference between the value of the composite feature vector of the monitored session and the entire user. At this time, the composite property vector value of all users is updated by every HTTP request that occurs in the web server, and the vector value of the monitored session is updated by the HTTP request that occurs in the session.

In the present invention, seven differentiation characteristics (Client-Server Bytes, Head request rate, Referer page existence rate, HTTP Error rate, Image resource and dynamic / Static web page rate, Inter-Request Gap time average, and 'robots.txt' file request) are selected. The above properties can be easily calculated from the Internet Information Service (IIS) standard extension format, so that they can be applied to various web servers using the IIS standard format. Not all seven differentiating characteristics show a great difference between a normal user and a web robot. Certain web robots show different values from normal users for some of the above characteristics, but similar values for other characteristics.

In the present invention, weight is given to a characteristic having a different value rather than a characteristic having a value similar to that of a normal user, and the reliability is increased by reflecting the degree of difference.

1 shows an overall configuration diagram of a system for detecting a web robot.

The system for detecting a web robot is generally composed of a web browser 110, a monitored web server 111, a web log server 112, a monitoring system 113, and a web robot detection server 200.

Web robot detection server 200 is composed of a data processing and pre-processing unit 220, detector 230, warning and reporting unit 330, the detection unit 230 is a whole user management unit 131, individual session management unit 132, the complex characteristic vector calculation unit 133 and the comparison analysis unit 134 of the monitored session.

The at least one web server 111 to be monitored means a plurality of web servers distributed on the Internet to be monitored according to a predetermined policy.

The web log server 112 collects a web log generated for each HTTP request requested from the user's WWW browser from the at least one monitored web server 111, and the web robot detection server The data processing and preprocessing unit 220 of 200 transmits a standard web log in raw format.

At this time, one HTTP request generates one web log, and the web logs are transmitted and processed in real time in order. The standard web log in raw format transmitted from the web log server 112 is processed into data in a usable form through the data processing and preprocessing unit 220 of the web log detection server 200 to detect the web robot. It is delivered to the detector 230 of the server 200.

The web robot detection server 200 collects standard web logs in a raw format that are generated for each HTTP request from the web log server 111 and stores them in a log queue, and repeatedly times out an HTTP session. In consideration of the first HTTP request generated, the web logs are parsed, and the differentiated properties for various web robots are extracted from the parsed information. Based on the relative differences of the polar coordinate system, it is represented by the angle and length in the complex characteristic vector format of the polar coordinate system, the session type is determined by the angle component, and the reliability of the discrimination is determined by the length component. Provided to the monitoring system 113.

The monitoring system 113 receives the detected result indicating whether the web robot or the normal user from the web robot detection server 200 to respond to the administrator.

The data processing and preprocessing unit 220 collects the web logs generated for each HTTP request from the web log server 112 and stores them in a log queue, and repeatedly first considers the timeout of the HTTP session. Parse web logs from generated HTTP requests, and define differentiation characteristics (Client-Server Bytes, Head request rate, Referer page existence rate, HTTP Error rate, Image resource and dynamic / static web page rate, Inter) -Request Gap time average, 'robots.txt' request or not) information is processed into a predetermined property array.

The detection unit 130 extracts individual monitoring target property information and total user average property information from a newly processed web log generated according to one HTTP request, and detects the current monitoring target session (HTTP session) and the entire user. The property ratio is calculated, normalized to the angle and length of the composite attribute vector (CAV) of the polar coordinate system, and the vector sum of the composite property vector (CAV) of the monitored sessions for all normal users and previous HTTP requests. It is updated, and it considers various characteristics in the form of complex characteristic vector of one polar coordinate system to determine the session type by angular component, guarantees the reliability of identification by length component, and detects whether it is a web robot or normal user. Results.

The detector 230 of the web robot detection server 200 includes an entire user manager 131, an individual session manager 132, a complex feature vector calculator 133 of a monitored session, and a comparison analyzer 134.

The whole user manager 131 manages all user average characteristic information.

The individual session manager 132 manages the characteristic information of the individual monitored session.

The complex characteristic vector calculation unit 133 of the monitored session extracts the characteristic information of the individual session to be monitored (HTTP session) and the average characteristic information of all users from the newly processed web log generated according to the one HTTP request. The composite component vector (CAV) between the current monitored session and all users is calculated, and the angle component using the relative difference between the average of the current HTTP session and all users from the web log according to one HTTP request ( α) and length component (β) are normalized, and the normalized angular component and length component are multiplied by the weights of the length components, taking into account the weighted average, and integrated into the composite characteristic vector values to obtain the angle and length of the composite characteristic vector And the vector sum of the CAV values for one HTTP request and the CAV values of all existing monitored sessions. And updates the value of the CAV.

Based on the CAV value reflecting the result of the last HTTP request, the comparison analysis unit 134 is configured in the angular region defined on the polar coordinate system based on the relative difference γ of the composite characteristic vector value of the current monitored session and the entire user. Accordingly, it is determined whether or not the web robot, the reliability of the result by the length component, and whether the web robot or normal user to determine the detection result and provides to the warning and reporting unit 330.

The warning and reporting unit 330 reports the detected result indicating whether the web robot or the normal user is from the comparison analysis unit 134 to the monitoring system 113 according to a predetermined policy of the monitoring target web server 111 to the administrator. In charge of.

FIG. 2 illustrates a data processing and preprocessing process in a data processing and preprocessing unit of a web robot detection server as a preliminary step for detecting a web robot.

Since raw data is provided from a standard web log provided from the weblog server 112, it is necessary to convert it into a form usable in the present invention.

The data processing and preprocessing unit 220 stores a standard web log provided from the weblog server 112 in a log queue (step 221), and then stores the standard web log in a log queue and repeatedly performs it. In step 222, the first HTTP request is searched for in consideration of a time out of the HTTP session. The first reason for looking for incoming requests is to consider the timeout of the HTTP session. The HTTP request is processed into information for obtaining seven differentiation characteristics used in the present invention through a web log parsing process (step 223) (step 224).

The data processing and preprocessing unit 220 includes information for obtaining seven differentiation characteristics for each HTTP request, namely, Client-Server Byte, Head Request, presence or absence of Referer field information, presence or absence of HTTP Error, Type ratio of request resources, converted into seconds (sec) or microseconds (msec), which is a standard unit of request time for obtaining inter-request gaps, is processed into information on whether there is a request for 'robots.txt' resource. The data processing and preprocessing unit 220 stores the result obtained in step 224 in the predetermined characteristic array and provides the data to the detector 230 (step 225).

FIG. 3 is a diagram illustrating a process of calculating and updating a composite attribute vector (CAV) of a whole user and a monitored session by receiving data preprocessed by the data processing and preprocessing unit of the web robot detection server.

Web robot detection server 200 according to the present invention is repeated for processing, updating, detection process for every HTTP request (HTTP Request). The data processing and preprocessing unit 220 of the web robot detection server 200 transmits a preprocessing result for the new HTTP request to the detection unit 230.

The detector 230 updates the average data of differentiation characteristics of the individual monitoring target session and the entire user with the differentiation characteristic data transmitted from the data processing and preprocessing unit 220 (step 231). The update is performed using EMA (Exponential Moving Average), counts, and averages for each differentiation characteristic information. The relative difference γ between the average of all the updated users and the average of the session to be monitored is calculated, and the angle component α and the length component β are normalized and calculated using the result (step 232).

The angular component is standardized to have a certain area. The larger the positive value is, the more similar the pattern is to the web robot, and the smaller or smaller the number is. In step 323, the angular component and the length component of each standardized characteristic are integrated into the angle θ and length ι of the CAV. Since it is determined to be similar to the web robot / normal user toward both ends of the angular component, the angular component is integrated into the angle of the CAV using a weighted mean that gives more weight toward both ends of the angular component.

The length is based on the sum of angle and length components of the calculated CAV, and as the current request shows a pattern similar to that of an ideal normal user or an ideal web robot, the weight is given to the sum of the length components. By multiplying the weights, it is designed to increase the impact of the request with the ideal result on the detection result.

The CAV value produced by one HTTP request and the CAV value of the monitored session processed up to the previous request are updated through a vector sum (step 234).

It determines whether it is a web robot or normal user based on the CAV value reflecting the result of the last HTTP request. The detected result is delivered to the alerting and reporting unit 330, which is then processed according to the routine (step 235).

The web robot detection method collects web logs generated for each HTTP request from at least one or more monitored web servers 111 and collects standard web logs in raw format from the web log server 112. Transfer to the detection server 200.

The web log server 112 collects the standard web logs in a raw format, which is generated for each HTTP request, from the web log server 112 to the web robot detection server 200, and stores them in a log queue. Considering the first HTTP request generated by parsing, we parse the weblog, extract the pre-defined differentiation characteristics from the parsed information, and calculate the ratio of the characteristics between the current session and all users. The result is provided to the monitoring system 113.

The differentiation characteristic information includes Client-Server Bytes, Head Request Rate, Referer Page Presence Rate, HTTP Error Rate, Image Resource and Dynamic / Static Web Page Rate, Time Average of Inter-Request Gap, Request of 'robots.txt' File. Includes whether information.

The present invention provides a single vector value that can be used to represent a normal user and a web robot by combining a number of characteristics. It is provided in a vector, with the angle determining whether the surveillance session is a web robot or a normal user. If the request with similar results persists, the length of the vector is increased by the vector sum, so the length indicates the reliability of the detection result. In the present invention, by reflecting the results of the area and length shown in FIG. 4 based on these results, the web-server administrator can help to take appropriate measures by warning and reporting.

4 shows an area for determining a detection result from a composite attribute vector (CAV). In the present invention, an error within -25 to + 25% is recognized as (413) based on all normal users, and when the error is +25 to + 75%, it is very likely that the web robot is (410). If the error is +75 to + 100%, the web robot of 411 is certain, and if the error is -25 to -75%, the web robot of (414) is slightly different from the overall user average. Defined as normal user area. 412 is a case opposite to the web robot and at the same time excessively larger than the overall user average. Although it rarely occurs in reality, it is not an obvious web robot, but is classified as an abnormal web robot because it is an abnormal user. In general, areas 410, 411, and 412 are defined as web robot areas, and 413 and 414 are normal user areas.

5 is a flowchart illustrating a web robot detection method according to the present invention.

The web log server 112 collects web logs generated for each HTTP request from the at least one or more monitored web servers 111 (step S10).

The data processing and preprocessing unit 220 of the web robot detection server 200 is a data processing and preprocessing unit 220 of the web robot detection server 200 from the web log server 112 for each HTTP request. Receives generated web logs and stores them in log queue, parses web logs from the first HTTP request repeatedly, extracts 7 different characteristics for various web robots from parsed information, and parses them The processed information is processed into differentiated characteristic information, and then stored in a predetermined characteristic array (step S11).

From the newly processed web log generated according to one HTTP request received from the data processing and preprocessing unit 220, the entire user management unit 131 and the individual session management unit 132 each individually monitor target property information and total user average. The characteristic information is extracted, and each characteristic information is provided to the complex characteristic vector calculation unit 133 of the session to be monitored (step S12).

The composite characteristic vector calculation unit 133 of the monitored session calculates a composite characteristic vector (CAV) between the current monitored session and all the users, and the current monitored session (HTTP session) and the whole from the web log according to one HTTP request. The normalized angle component and the length component using the relative difference from the user's mean are integrated into the composite characteristic vector values through the weight mean and the length weight, and the angle (θ) and the length ( ι) (step S13), and update the CAV value of the current monitored session through the vector sum of the CAV values for one current HTTP request and the CAV values of the entire monitored session up to the previous previous request including the same ( Step S14).

The comparison analysis unit 134 performs polar coordinates based on the relative difference between the composite characteristic vector values of the monitored session and the entire user based on the CAV value reflecting the result of the last HTTP request from the composite characteristic vector calculation unit 133 of the monitored session. Indicate the angle and length of the complex characteristic vector on the system, determine the session type by the angle component, check the reliability of the web robot identification by the length component, determine whether it is a web robot according to the angle region, and detect the result by the length component. The reliability is verified (step S15).

The comparison analysis unit 134 of the detection unit 230 determines and determines whether the detection result is a web robot or a normal user, and provides the detected result to the warning and report unit 330 (step S16).

The warning and reporting unit 330 of the web robot detection server 200 provides the monitoring system 113 with the detected result, whether it is a web robot or a normal user (step S17).

The web robot detection system according to the present invention comprises: (a) collecting, by the web log server, a web log generated for each HTTP request from the at least one or more monitored web server by the web log server, and raw from the web log server; Receiving a standard web log in a format (Raw Format) from the web robot detection server; (b) receiving a web log generated for each HTTP request from the web log server to a data processing and preprocessing unit, parsing the web log from the first HTTP request generated, and providing a predetermined differentiation characteristic of the parsed information; function; (c) The complex characteristic vector calculation unit of the monitored session extracts the characteristic information of the individual monitored session and the average characteristic information of all the users from the newly processed web log generated according to the one HTTP request, and is currently monitored. Calculate a composite property vector (CAV) between the session and all users, normalize and calculate the angular and length components using the relative difference between the current HTTP session and the average of all users from a web log of one HTTP request, The normalized angular component and the length component are multiplied by the weight by taking the weighted average into consideration and integrated into the composite feature vector values to calculate the angle and length of the composite feature vector (CAV). Updating the CAV value of the monitored session through the vector sum of the CAV values of all existing monitored sessions including; (d) The comparison analysis unit displays the polarity based on the relative difference between the composite feature vector values of the monitored session and the entire user based on the CAV value reflecting the result of the last HTTP request from the composite feature vector calculation unit of the monitored session. Determining whether a web robot is determined according to a defined angular region, checking reliability of a result by a length component, and determining whether the web robot is a normal user or providing the detected result to an alerting and reporting unit; And (e) a computer-readable recording medium having recorded thereon a program for realizing a function of determining whether the web robot is a normal user or a normal user from an alerting and reporting unit and providing the detected result to a monitoring system.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, etc.) in a form readable by software of a computer.

As described above, the present invention has been described in detail with reference to preferred embodiments, but the present invention is not limited thereto, and various changes and applications may be made without departing from the technical spirit of the present invention. Self-explanatory Accordingly, the true scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of the same should be construed as being included in the scope of the present invention.

110: WWW Browser 111: Watched Web Server
112: web log server 113: monitoring system
200: web robot detection server 220: data processing and preprocessing unit
230: detection unit 231: total user management unit
232: individual session management unit 234: comparison operation unit
233: Complex attribute vector calculator of the monitored session
330: warning and reporting

Claims (10)

In the web robot detection system,
At least one monitored web server connected to the Internet;
A web log server which collects a web log generated for each HTTP request from the at least one monitored web server and delivers a standard web log in raw format;
By collecting the standard web log of the raw format form generated for each HTTP request from the web log server, taking into account the differentiation characteristic information extracted from the standard web log of the complex attribute vector (CAV: Composite Attribute Vector) A web robot detection server configured to display angles and lengths to provide detected results indicating whether the web robot is a normal user; And
A monitoring system for receiving a detected result indicating whether a web robot or a normal user is received from the web robot detection server;
Web robot detection system comprising a. The method of claim 1,
The web robot detection server,
Collect web logs generated for each HTTP request from the web log server, store them in a log queue, parse the web logs from the first HTTP request generated repeatedly in consideration of the timeout of the HTTP session, A data processing and preprocessing unit which processes the parsed information into differentiation characteristic information and stores the same in a predetermined characteristic array;
From the newly processed web log generated according to the one HTTP request, the characteristic information of the individual monitored session and the average characteristic information of all the users are extracted, and the ratio of the characteristic between the current monitored session (HTTP session) and all users A detection unit for calculating the detection result by indicating the angle and the length of the complex feature vector (CAV) and determining a detection result to provide a detected result indicating whether the web robot is a normal user; And
A warning and reporting unit for reporting the detected result indicating whether the web robot or the normal user is from the detection unit to the monitoring system according to a predetermined policy of the monitoring target web server to the administrator;
Web robot detection system comprising a. The method of claim 2,
The detection unit,
An overall user manager for managing all user average characteristic information;
An individual session manager configured to manage characteristic information of the individual monitored session;
Extracting individual monitored property information and total user average property information from a newly processed web log generated according to the one HTTP request, calculating a composite property vector (CAV) between the current monitored session and all users, and From the web log according to one HTTP request, the angle component (α) and the length component (β) are normalized and calculated using the relative difference (γ) between the current monitored session (HTTP session) and the average of all users. Multiply the length component by the weight, taking into account the weighted average, and calculate the angle and length of the composite characteristic vector (CAV), and the CAV value for one HTTP request and the CAV of the entire existing session to be monitored. A complex characteristic vector calculation unit of the monitored session for updating the CAV value of the current monitored session through the vector sum of the values; And
Based on the CAV value reflecting the result of the last HTTP request from the composite characteristic vector calculation unit of the current monitored session, the angle region defined on the polar coordinate system based on the relative difference between the composite monitored vector value of the current monitored session and all users. A comparison analysis unit to determine whether the web robot according to the verification, the reliability of the result by the length component, and determine whether the web robot or normal user to provide the detected result;
Web robot detection system comprising a. The method of claim 1,
The differentiation characteristic information,
Client-Server Bytes, Head Request Rate, Referer Page Presence Rate, HTTP Error Rate, Image Resource and Dynamic / Static Web Page Rate, Time Average of Inter-Request Gap, Request Information of 'robots.txt' File Web robot detection system characterized. The method of claim 3,
The detection unit,
The average data of the differentiated characteristics of the individual monitored session and the entire user is updated using the EMA (Exponential Moving Average), the count, and the average with the differentiated characteristic information transmitted from the data processing and preprocessing unit, and the updated total is updated. A web robot detection system comprising calculating a relative difference γ between an average of a user and an average of the current monitored session, and normalizing and calculating an angle component α and a length component β using the result. . In the web robot detection method comprising a web log server, a web robot detection server, and a monitoring system linked to at least one web server to be monitored,
(a) collecting web logs generated for each HTTP request from the at least one or more monitored web servers to a web log server; And
(b) Parse the web logs from the HTTP request by collecting standard web logs in the raw format generated for each one HTTP request from the web log server to the web robot detection server, and match the various web robots from the parsed information. Based on the relative difference between the composite property vector values between the current monitored session (HTTP session) and all users, the differentiation characteristics are extracted and expressed in angle and length in the form of complex feature vectors in the polar coordinate system. Providing the detected result to a monitoring system;
Web robot detection method comprising a. The method of claim 6,
The step (b)
(b1) collecting the web logs generated for each HTTP request from the web log server to the data processing and preprocessing unit of the web robot detection server, storing them in a log queue, and repeating the first HTTP request Parsing a weblog, processing the differentiated characteristic information from the parsed information, and storing the weblog in a predetermined characteristic array;
(b2) the complex property vector computing unit of the current monitored session (HTTP session) extracts the individual monitored object property information and the total user average property information from the newly processed web log generated according to the one HTTP request; Compute a composite characteristic vector (CAV) between the current monitored session and all users, and calculates the angle using the relative difference (γ) of the average of all updated users in the current monitored session from the web log according to one HTTP request. Compute normalized component (α) and length component (β), calculate the angle and length of the composite characteristic vector (CAV) by applying weighted average and length weighted normalized angular component and length component, The CAV value of the current monitored session is updated by the vector sum of the CAV values for the request and the CAV values of the entire monitored sessions up to the previous previous request. Step;
(b3) The comparative analysis unit calculates the relative difference between the composite feature vector values of the monitored session and the current monitored session of all users based on the CAV value reflecting the result of the last HTTP request from the composite feature vector calculation unit of the monitored session. Determining whether the robot is a web robot according to the angular region defined on the polar coordinate system, checking the reliability of the result by the length component, and determining whether the web robot is a normal user or providing the detected result to the warning and report unit; And
(b4) determining whether the web robot is a normal user from the warning and reporting unit and providing a detected result to a monitoring system;
Web robot detection method comprising a. The method of claim 7, wherein
The differentiation characteristic information,
Client-Server Bytes, Head Request Rate, Referer Page Presence Rate, HTTP Error Rate, Image Resource and Dynamic / Static Web Page Rate, Time Average of Inter-Request Gap, Request Information of 'robots.txt' File Web robot detection method characterized in that. The method according to claim 7 or 8,
Step (b2),
The average data of the individual monitored sessions and the total user's differentiation characteristic information is updated by using the EMA (Exponential Moving Average), counts, and averages with the differentiated characteristic data transmitted from the data processing and preprocessing units, and the average of the updated total users. And calculating the relative difference (γ) between the average of the monitored sessions and using the result to normalize the angular component (α) and the length component (β). (A) a function of collecting a web log generated for each HTTP request from the at least one or more monitored web server to a web log server;
(b) The web processing server of the web robot detection server receives the web log generated for each HTTP request from the web log server and stores the web log in the log queue, and repeatedly parses the web log from the first HTTP request. And processing the differentiated characteristic information from the parsed information and storing the same in a predetermined characteristic array;
(c) The complex characteristic vector calculation unit of the monitored session extracts the characteristic information of the individual monitored session and the average characteristic information of all the users from the newly processed web log generated according to the one HTTP request, and is currently monitored. Calculate a composite property vector (CAV) between the session and all users, normalize and calculate the angular and length components using the relative difference between the current HTTP session and the average of all users from a web log of one HTTP request, The normalized angular component and the length component are multiplied by the weight component by considering the weighted average to calculate the angle and length of the composite characteristic vector (CAV), and the CAV value for one current HTTP request and the previous previous request including the same. Updating the CAV value of the current monitored session through a vector sum of the CAV values of the entire monitored session;
(d) The comparison analysis unit displays the polarity based on the relative difference between the composite feature vector values of the monitored session and the entire user based on the CAV value reflecting the result of the last HTTP request from the composite feature vector calculation unit of the monitored session. Determining whether a web robot is defined according to a defined angular region, checking reliability of a result by a length component, and determining whether the web robot is a normal user or providing a detected result to an alerting and reporting unit; And
(e) a function of determining whether the web robot is a normal user from a warning and report unit and providing a detected result to a monitoring system;
A computer-readable recording medium that records a program for realizing the problem.

Images