KR101556714B1 - Method, system and computer readable recording medium for providing search results - Google Patents

Abstract

The present invention relates to a method for clustering a source document, comprising: clustering a source indication clustering a web document source indication corresponding to a web document stored in a database into predetermined criteria; A similar document clustering step of grouping similar documents into the source indication clustering result; Classifying variables using features generated from the source indication clustering step and the similar document clustering step, respectively; Generating a regular expression using the result of the variable classifying step; And collecting the web document by using the generated regular expression. The present invention relates to a method and system for providing a search result.

Description

[0001] METHOD, SYSTEM AND COMPUTER READABLE RECORDING MEDIUM FOR PROVIDING SEARCH RESULTS [0002]

The present invention relates to a method and system for constructing a search database by removing unnecessary variables from the source display of a web document such as a URL (Uniform Resource Locator), and more particularly, To a method, system and computer readable medium for constructing a search database using a regular expression pattern in which unnecessary variables are automatically extracted and removed.

In recent years, as the use of the Internet has become popular, users can acquire various information through the Internet search. That is, users access an Internet search site by inputting an identifier such as a URL in an address window of a web browser through a terminal device such as a personal computer capable of accessing the Internet, input a search phrase to be confirmed by themselves, , Games, communities, web documents, and more.

In order to appropriately display the contents to be searched by users, the Internet search site provider collects web documents and the like, constructs an index for the collected web documents, and provides search results to the users based on the indexes And web crawler, which functions to search and collect web documents existing on the Internet in an organized and automated way, plays a big role.

One of the ways in which the web crawler works is to start from a list of URLs, usually called seeds, to recognize all hyperlinks included in the seed, update the URL list, As well as to visit the site again.

On the other hand, at the time of web page production, the web creator does not change the contents of the page in order to manage the functions of the web page, but it is possible to add variables having various values to the URL. And collecting all of these same web documents in contents, it is possible to collect unnecessary or meaningless web documents in a duplicate manner, resulting in a waste of storage space for storing the collected contents, and the performance of the web crawler is deteriorated , And the load of the search engine may be increased.

Accordingly, in order to solve such a problem, conventionally, an administrator manually inputs a regular expression pattern for removing such unnecessary variables in a URL, and deletes a duplicated web document among the previously stored web documents. In addition, when the web crawler extracts a URL to visit a new document, it has avoided unnecessary page visits by applying this regular expression pattern and storing the URL. However, due to the limitation of manually entering regular expression patterns, there is still a problem that is difficult to cover substantially the entire massive web.

An object of the present invention is to solve the above-mentioned problems of the prior art.

One object of the present invention is to efficiently utilize the storage space of the search database and reduce the load of the search engine by automatically determining and extracting unnecessary variables in the web document source display to delete redundant ones of the previously stored web documents .

It is another object of the present invention to provide a method and system for automatically collecting unnecessary variables in a web document source display and generating a regular expression for eliminating the unnecessary variables and applying the same to a web crawler.

In order to achieve the above-described object of the present invention and to achieve the specific effects of the present invention described below, the characteristic structure of the present invention is as follows.

According to an aspect of the present invention, there is provided a method for providing a search result corresponding to a search query term of a user,

CLAIMS 1. A method for clustering a source document, the method comprising: clustering a source indication clustering a source document indication corresponding to a web document stored in a database;

A similar document clustering step of grouping similar documents into the source indication clustering result;

Classifying variables using features generated from the source indication clustering step and the similar document clustering step, respectively;

Generating a regular expression using the result of the variable classifying step; And

And collecting the web document using the generated regular expression.

In addition, preferably, the web document source display includes a URL (Uniform Resource Locator), and the source display clustering step may include a step of, when the URLs are the same from the URL to the URLs of the web document, , Or the same up to the parameter key (key), it can be grouped into the same cluster.

Also, preferably, the similar document clustering step may be performed on the web document source display lists enclosed in the same cluster by the source display clustering step.

In addition, preferably, the similar document clustering step may perform similarity determination on all pairs of web document source display in the same cluster.

In addition, preferably, the similar document clustering step includes:

Parsing the web document;

Hashing the parsed web document;

Calculating a simhash by computing the hashed web document bit by bit; And

And grouping the web documents whose hamming distance is equal to or less than a predetermined value as a similar document.

In addition, preferably, the feature generated from the source indication clustering step may include at least one of a parameter key related feature and a parameter value related feature.

Preferably, the feature generated from the similar document clustering step includes a feature relating to a ratio of a unique value of a corresponding variable to a total source indication cluster count when creating a similar document pair, A feature associated with the ratio of the unique document involved in creating the document, a feature associated with the unique document versus the number of similar document pairs, and a feature associated with the absolute value of the similar document pair count.

In addition, preferably, the step of classifying the variables includes classifying categories for each variable contained in the web document source indication grouped into one similar document cluster, and classifying the classified categories into variables for managing the user's access session And classifying the variable as an unnecessary variable when the category corresponds to a variable category for checking where the document or the web document is linked.

In addition, the step of generating the regular expression may generate a regular expression pattern that removes the unnecessary variable from the target web document source display.

According to another aspect of the present invention, there is provided a system for providing a search result corresponding to a search query term of a user,

A source indication clustering means for grouping, based on a predetermined criterion, web document source indication corresponding to a web document stored in a database;

Similar document clustering means for clustering similar documents with respect to the source indication clustering result;

Variable classifying means for classifying a variable using features generated from the source indication clustering means and the similar document clustering means, respectively; And

A regular expression applying means for generating a regular expression using the result from the variable classifying means; A similar document removing unit including the similar document removing unit,

And a search unit for collecting the web document using the generated regular expression.

Preferably, the web document source indication includes a URL (Uniform Resource Locator), and the source indication clustering means comprises:

The URLs of the web document may be grouped into the same cluster when the URLs are the same up to the path, the file names are the same, or the parameter keys are the same.

In addition, preferably, the similar document clustering means may perform similar document detection on the Web document source display lists enclosed in the same cluster by the source display clustering means.

In addition, preferably, the similar document clustering means may perform similarity determination on all pairs of web document source display in the same cluster.

Further, preferably, the similar document clustering means comprises:

Parsing a web document;

Hashing the parsed web document;

Computing the hash web document bit by bit to calculate a simhash; And web documents whose hamming distance is equal to or less than a predetermined value can be grouped as a similar document.

Also preferably, the feature generated from the source indication clustering means may comprise at least one of a parameter key related feature and a parameter value related feature.

Preferably, the feature generated from the similar document clustering means comprises a feature relating to a ratio of a unique value of the variable to the total source indication cluster count when creating a similar document pair, A feature associated with the ratio of the unique document involved in creating the document, a feature associated with the unique document versus the number of similar document pairs, and a feature associated with the absolute value of the similar document pair count.

Preferably, the variable classification means classifies categories for each variable contained in the web document source indication grouped into one similar document cluster, and classifies the category classified into a variable category for managing the user's access session or If the variable category is checked to check where the web document is linked, the variable can be classified as an unnecessary variable.

In addition, preferably, the regular expression applying means may generate a regular expression pattern for removing the unnecessary variables from the target web document source display.

As described above, according to the present invention, unnecessary variables are automatically determined and extracted in the web document source display, thereby eliminating redundant ones of the web documents stored in the past, thereby increasing the use efficiency of the storage space of the search database, .

In addition, according to the present invention, it is possible to provide a function of collecting a web document more efficiently in the future by generating a regular expression for automatically determining an unnecessary variable and removing the unnecessary variable from the display of the web document source.

FIG. 1 is a diagram schematically showing the overall configuration of a search result providing system using a search database constructed by collecting web documents after automatically removing unnecessary variables from URLs according to an embodiment of the present invention.
2 is a detailed configuration diagram of a search result providing system according to an embodiment of the present invention.
3 is a detailed configuration diagram of a similar document removing unit in a search result providing system according to an embodiment of the present invention.
4 is a view for explaining the concept of an NDD cluster according to an embodiment of the present invention.
5 is a flowchart illustrating a search result providing method according to an embodiment of the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which the claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily carry out the present invention.

In an embodiment of the present invention, the term "connection " means broad meaning, including not only when it is" directly connected "but also when" electrically connected " Also, when an element is referred to as "comprising ", it means that it can include other elements as well, without departing from the other elements unless specifically stated otherwise.

Further, in the embodiment of the present invention, the term "web document" should be interpreted in a broad sense including both manual and active document formats that can be browsed using a web browser program such as Internet Explorer, Although HTML (HyperText Markup Language) is mainly used as a document format, it is not necessarily limited to this, and any document format including XML (Extensible Markup Language) and SGML (Standard Generalized Markup Language) It corresponds to a web document. In order to browse a web document using a web browser program, an address in which a web document is located is generally input as a URL, and HTTP (HyperText Transfer Protocol) is often used as the address format. However, the present invention is not limited thereto.

On the other hand, in the specification of the present invention, the term " web document source display "includes characters, directives, and the like expressed in a predetermined manner so as to know the source of the web document including URLs to be described later.

In addition, the term "URL (Uniform Resource Locator)" is used to specify the location of files on each server that provides services on the web. The type of service to be accessed, the location . The general syntax of a URL can be configured in the same format as "protocol: // hostname / path / filename? Parameter". The path may include a plurality of paths, and the parameter may include a plurality of parameters. For example, in a URL such as http://www.naver.com/a/b/c.html?x=1&y=2, the protocol is http, the host name is www.naver.com, the path is / a / b /, the file name is c.html, and the parameters are x and y, and the values are 1 and 2, respectively.

When collecting web documents through web crawlers, if the URLs are slightly different and collect all of them, for example, only web pages with different counters such as counters only or calendars other blogs, Quot; similar document ") are stored redundantly, and thus a large number of meaningless web documents collected are deleted from the storage space such as a database again. Therefore, in the present invention, a regular expression for automatically extracting and deleting unnecessary variables from a URL Discloses a search system and method for excluding a meaningless web document from a collection.

Complete system configuration

FIG. 1 schematically shows a general configuration of a search result providing system using a search database constructed by collecting web documents after automatically determining and removing unnecessary variables from a URL through a similar document detection method according to an embodiment of the present invention. Fig.

1, an overall system according to an exemplary embodiment of the present invention includes a search result providing system 100 for providing a plurality of user terminal devices 300 and a plurality of web document servers 400).

First, according to an embodiment of the present invention, the search result providing system 100 receives a search phrase, i.e., a query term, from the user terminal device 300 and performs a search by referring to the search database (not shown) And transmits the retrieved search result to the user terminal device 300. In addition, the search result providing system 100 may use a web crawler (refer to 140 in FIG. 2) to prevent redundant collection of similar documents from web documents collected from one or more web document servers 400, And deletes the similar document by applying it to the already stored search database, or may apply the regular expression generated in the future operation of the web crawler (not shown).

In addition, according to one embodiment of the present invention, the communication network 200 may be configured without regard to its communication mode such as wired and wireless, and may be a personal area network (PAN), a local area network , A metropolitan area network (MAN), a wide area network (WAN), and the like.

Meanwhile, the user terminal 300 according to an exemplary embodiment of the present invention includes a function for connecting with the search result providing system 100 through the communication network 200 in order to receive a search result for a predetermined query term Means an input / output device and may be a memory device such as a desktop computer as well as a mobile communication terminal including a notebook computer, a workstation, a palmtop computer, a personal digital assistant (PDA), a web pad, And can be adopted as the user terminal device 300 according to the present invention as long as it is a digital device having computation capability by mounting a microprocessor. Preferably, the web browser in the user terminal device 300 may be operated and used in connection with the search result providing system 100, and in order to receive a search result by inputting a query word, the present invention is not limited thereto.

The web document server 400 according to an embodiment of the present invention is collectively referred to as a web server including a web document to be collected by the web crawler in the search result providing system 100, Format web documents. The web server that can access the web crawler through the communication network 200 and collect the web document is all included in the web document server 400.

Search result providing system

2 is a detailed configuration diagram of a search result providing system 100 according to an embodiment of the present invention.

Referring to FIG. 2, a search result providing system 100 according to an embodiment of the present invention includes a search unit 110, a similar document removing unit 120, a web crawler 130, and a search database 140 .

The search unit 110 searches the search database 140 for information matched with the query term received from the user terminal device 300. The search result extracted through the search is transmitted to the user terminal device 300.

The similar document removing unit 120 deletes the similar documents from the web document server 400 through the web crawler 130 in order to prevent duplicate search of similar documents whose URLs are partially different but almost similar in contents A regular expression for automatically extracting and deleting unnecessary variables is generated and applied to the web document stored in the search database 140 by applying the generated regular expression and predetermined conditions so as to delete the redundantly stored similar document have. In addition, the similar document removal unit 120 can set the web crawler 130 so that the similar crawler 120 does not redundantly collect the similar document based on the regular expression in the operation of the web crawler 130 in the future. Detailed function of each component of the similar document removing unit 120 will be described later.

The web crawler 130 searches and collects the web documents stored in the web document server 400 in a known systematic and automated manner and stores them in the search database 140 or in a separate database. In addition, according to one embodiment of the present invention, the web crawler 130 uses the regular expression provided in the similar document removing unit 120, thereby, unlike the prior art in search and collection of web documents, It will not be collected.

The search database 140 may include various information collected or stored to provide a search result corresponding to a query term, and may also store a web document collected by the web crawler 130. In addition, if a similar document which has almost no difference in content from the web document already searched in the search database 140 is redundantly stored by the operation of the similar document removing unit 120, some or all of the similar document may be deleted .

Although only the search database 140 is shown in FIG. 2, according to an embodiment of the present invention, a database for storing web documents collected and detected by the web crawler 130 is separately constructed, Indexed and stored in the search database 140. Although the search unit 110, the similar document removing unit 120, and the web crawler 130 are shown as separate blocks in the drawing, they may be physically implemented in one machine, or some or all of them may be physically It may be implemented in another machine, or a plurality of physically existing machines having the same function may exist in parallel. It will be apparent to those skilled in the art that various modifications and changes may be made in the present invention without departing from the spirit and scope of the invention.

Similar Documents Removal

3, the similar document removing unit 120 in the search result providing system 100 according to the embodiment of the present invention will be described in more detail. The similar document removing unit 120 includes a URL clustering unit 121, A near duplication detection (NDD) clustering means 122, a variable classifying means 123, and a regular expression applying means 124.

The URL clustering unit 121 according to an embodiment of the present invention collects the URLs of the web documents collected by the web crawler 130 stored in the search database 140 or a separate database in the same cluster unit, It can analyze the URLs included in the same cluster and find the related features.

Specifically, the URL clustering can be performed through various methods. For example, if the URLs are the same from the file names, the clustering may be performed in the same cluster, or the same parameter key may be regarded as the same cluster. Or may be viewed as the same cluster up to the same path. Alternatively, a URL cluster may be constructed by ignoring key and / or parameter values of a component or parameter of a path whose value is frequently changed in the corresponding domain.

Hereinafter, A to E in Table 1 show examples of URLs grouped in the same URL cluster.

URL A https://www.kokutou.com/products/review.php?PHPSESSID=14d3bd1d0e896 B https://www.kokutou.com/products/review.php?PHPSESSID=14d9f0ddfk6896 C https://www.kokutou.com/products/review.php?PHPSESSID=54d0h9c5n78hse D https://www.kokutou.com/products/review.php?PHPSESSID=d9h8v7e6d0f9h8 E https://www.kokutou.com/products/review.php?PHPSESSID=z0d98n7e6d7f98

In other words, according to the embodiment of the present invention, the URL clustering method may use any known method, and only the common information of the URLs bundled with the same URL cluster may be left as a feature, (123) can use it.

At this time, the URL clustering result extractable features may include at least one of, for example, a parameter key or a feature associated with a parameter value. However, the present invention is not limited thereto, and more various features may be extracted according to the URL clustering method.

First, the parameters associated with the parameter key include a key for each key value of the variable, a keyinclude related to whether a specific value is included in each parameter key value, a file name (Cgi) for the cgi value when the extension ends with cgi or php, and a feature associated with the previous path value before the filename including cgi or php (lastpath), but is not limited to no.

Parameter key related features Yes key ▶ key_sid, key_no, key_page keyinclude ▶ sid, sessid, session, phpsessid, sess, phpid, clientid, accessid, who
Redirect, return, url, f_file, fname, from, refer, token, src, login cgi ▶ cgi_board.cgi / cgi_login.php / cgi_bbs.php etc. lastpath ▶ lastpath_zboard / lastpath_bbs / lastpath_calendar etc.

Next, the features associated with the parameter values include, as illustrated in Table 3 below, the deviation of the length of the feature (vtype) associated with the type of the parameter value, the feature associated with the average length vlength of the parameter values, But may include, but is not necessarily limited to, at least one of the related features (vlengthdev).

Parameter Value Related Features Yes vtype ▶ vtype _blank: No value of variable
▶ vtype _integer: integer
▶ vtype _ip_addr: ip address
Vtype_mixed_number_alphabet: A string of numbers and alphabets randomly mixed. For example, if the number of times of number / character conversion is more than the predetermined number, abc1015 (corresponding x), a1b0c15 (corresponding o)
▶ vtype _too_long_value: string with a length greater than or equal to a predetermined value VLength ▶ vlength _20: The average length of the parameter values is less than 20
▶ vlength _20_30: 20 or more and less than 30 vlengthdev ▶ vlengthdev_fixed: When the deviation value of the length of the parameter values is less than the predetermined value
▶ vlengthdev_diverse: If the deviation value is more than the predetermined value

Therefore, if we look at the features that can be generated from the URL clusters A to E illustrated in Table 1, keyinclude, cgi, and lastpath can be generated as parameter key related features, and the values correspond to PHPSESSID, review.php, and products / do. In addition, vtype, vlength, and vlengthdev can be generated as parameter value related features, and the values correspond to vtype_mixed_number_alphabet, vlength_20, and vlengthdev_fixed, respectively. A method of classifying variables using the features extracted from URL clusters and their values will be described later.

Meanwhile, the NDD clustering unit 122 according to an exemplary embodiment of the present invention may be configured such that the URL of the web documents collected by the web crawler 130 is partially different but the contents of the web document does not change and similar similar documents exist It is possible to perform the function of bundling the URLs of the web documents that have passed through the URL clustering means 121 into NDD clusters according to a predetermined criterion. Through this work, it is possible to greatly reduce the amount of work and web documents that need to collect similar documents redundantly, and thus the processing cost can be reduced and the processing speed can be improved.

That is, the NDD clustering means 122 performs similarity determination on all pairs of URLs in the same cluster, targeting the URL lists bundled in the same cluster as shown in Table 1, and groups similar document pairs It is classified into the same NDD cluster. For example, as illustrated in Table 1, NDD clustering for (A, B), (A, C), (A, D), All pairs of URLs in the same cluster of (B, C), (B, D), (B, E), (C, D), A determination can be made.

However, the NDD clustering according to the NDD clustering means 122 may be performed entirely for all the URLs passed through the URL clustering means 121 or for a part of the URLs passed through the URL clustering means 121 The normalized URL may be applied to the rest of the URL based on the normalized URL, and the change in the manner or order of operations will be apparent to those skilled in the art.

Specifically, the method of NDD clustering can handle large-scale documents through, but not limited to, hash-based similarity detection (hereinafter referred to as "simhash"). In addition, it is possible to measure the degree of similarity between web documents by calculating the shingles of each document, and calculating the Jaccard coefficient based on the shingles, It will be apparent to those skilled in the art that it is possible to quickly detect whether a similar document exists between documents. The similarity determination based on the deep sea method will be described below for ease of explanation.

In detail, each web document is divided into words by parsing, and the text nodes, anchor addresses, image source addresses, and embed tag addresses divided by word units are subjected to hashing (hashing), and then calculating it by bit to calculate the deep-sea time. By using the calculated deep sea scenes, web documents having a hamming distance of a predetermined value, for example, 3 or less, can be regarded as similar documents, and these web documents can be grouped together into NDD clusters. However, according to the embodiment, it is also possible to group NDD clusters on the basis of the case where the Hamming distance has a value other than 3. Therefore, the present invention is not limited to a specific Hamming distance value.

(A, B), (A, C) and (B, C) pairs are bundled as "NDD cluster 1" in the case of the URLs in Table 1, for example, as a result of NDD clustering through the above- , (D, E) can be bundled into "NDD cluster 2" (see FIG. 4). Next, the features can be extracted from each NDD clusters based on the NDD clusters thus bundled.

Specifically, as illustrated in Table 4 below, when creating an NDD pair, a feature (nddvalue) related to the ratio of the unique value of the variable to the total URL cluster count, an NDD pair relative to the entire URL cluster count At least one of a feature (ndddoc) related to the ratio of the only documents involved, a feature (nddcountmultiple) related to the number of unique documents versus the number of NDD pairs, and a feature (nddcountabs) related to the absolute value of the NDD fair count can be extracted Other features related to the similarity of web documents can also be extracted. It should also be apparent to those skilled in the art that the ratios determining the values of each of the features illustrated in Table 4 are not limiting, and that the ratios can be varied.

Features of NDD Clusters Yes nddvalue ▶ nddvalue_one: if there is one unique value
Nddvalue_small / nddvalue_medium / nddvalue_big: unique If there are two or more values, separated by small, medium, or big depending on the ratio size. ndddoc Ndddoc_one / ndddoc_small / ndddoc_medium / ndddoc_big nddcountmultiple ▶ nddcountmultiple_1: ratio (= number of unique documents / number of ndd pairs) is 1 or more and less than a predetermined value
▶ nddcountmultiple_10: ratio is more than 10 and less than predetermined value nddcountabs ▶ nddcountabs_10: absolute value is more than 10 and less than predetermined value
▶ nddcountabs_100: Absolute value is more than 100 and less than predetermined value

Therefore, in the above example, values for nddvalue, ndddoc, and nddcountmultiple, which are features extractable from the "NDD cluster 1" of the pairs (A, B), (A, C) and (B, C) are nddvalue_big, ndddoc_big, nddcountmultiple , And the values for nddvalue, ndddoc, and nddcountmultiple, which are features extractable from the "NDD cluster 2" of the (D, E) pair, are nddvalue_big, ndddoc_big, and nddcountmultiple_1, respectively. For the feature nddcountabs, the number of the population in Table 1 is small, and a separate explanation will be omitted. A method of classifying the parameters using the features extracted from the NDD clusters will be described later.

Next, the variable classifying unit 123 according to the embodiment of the present invention combines the URL clusters and the features extracted from the NDD cluster to determine whether a variable included in the URL is an unnecessary variable, and based on the determination It is possible to judge whether or not the URL of the similar document is the same as that of the web document already collected but only the URL is slightly different. In addition, the variable classifying unit 123 according to an embodiment of the present invention may further perform the function of providing whether a variable included in the URL belongs to which classification, based on the similarity and features among the web documents.

The variable classifying unit 123 can classify the URL clusters and the features extracted from the NDD cluster according to the characteristics of the corresponding variables based on various machine learning algorithms learned for the features of the correct set, It is possible to determine whether the variable is an unnecessary variable in the URL. The variable classification means 123 may be, for example, a classifier using a machine learning algorithm such as a Naive Bayesian classifier or Fisher's method. It should be understood, however, that the present invention is not limited to any particular classification technique or classifier.

Hereinafter, the operation of the variable classifying unit 123 according to an embodiment of the present invention will be described. The variable classifying unit 123 classifies the sample URL and a specific variable The similarity of the two web documents, URL clusters, combinations of features extracted from the NDD clusters, and combinations of the features extracted from the NDD clusters are grouped into one NDD cluster For each variable included in the URL, it is classified into categories such as sid, referrer, good, and bad as shown in Table 5 below. Here, the URL from which a specific variable is removed may refer to a URL that deletes a specific parameter value or deletes both a specific parameter key and a parameter value. The similarity of the two web documents may be determined by using the similarity judgment criterion Can be used.

Specifically, sid is a case in which two web documents are similar and the nature of the corresponding variable corresponds to a session ID, which is a variable for managing a user's access session, and referrer is a case in which two web documents are similar, Good is the case where the content of the two web documents is different depending on the presence or absence of the variable, that is, the document is not similar, and bad is the variable The content of the web document is not changed depending on whether or not the variable is a sid or a referrer. However, the bad variable may include, for example, a variable such as a calendar type variable or an unnecessary keyword, an email, a title, a sort, a page, and the like, May be extracted and added as an item corresponding to another similar document or the bad variable itself may be classified into a category different from good to remove the similar document more effectively. However, according to the embodiment, To avoid being excluded from web crawling, it is also possible to classify the variables of the bad category as a good category, that is, a non-native document. That is, the variable classifying unit 123 according to an embodiment of the present invention can classify corresponding variables included in a URL bundled into one NDD cluster using a URL of sid + referrer + (good + bad) as a correct set have.

category Yes sid Two document variations, variable personality = session ID referrer Two document variations, the nature of the variable = referrer good Both documents are different bad Two document variations, the nature of the variable ≠ session ID or referrer

If the similarity between the web document corresponding to the sample URL and the web document corresponding to the URL from which the specific variable is removed is below a certain value, the corresponding variable included in the URL bundled into one NDD cluster need not be considered in the good category . If the similarity degree of the web documents is more than a specific value, it is possible to determine the nature of the variable by combining various features. For example, if the keyinclude of the parameter key-related features has phpsessid, the personality of the variable corresponds to the session ID Since the probability is high, it can be weighted to be more likely to be included in the sid category. As described above, the variable included in the URL bundled into one NDD cluster may correspond to the sid, referrer, or bad category according to a value corresponding to a combination of various features having different weights.

In order to determine the most accurate weights for the various features and the most precise combination of features, the variable classification means 123 may use known machine learning algorithms, such as the Fisher's method, as described above, And the initial learning can be performed based on the classification. In addition, learning of the variable classifying means 123 is performed on the basis of the result of classifying the variables of some URLs collected periodically or irregularly from the already applied regular expression pattern to increase the accuracy thereof, It is possible to implement various methods for improving the accuracy, for example, by generating patterns of patterns and generating patterns, and by analyzing the accuracy and deleting patterns with low accuracy.

Accuracy experiment of variable classification according to the present invention

In order to confirm the accuracy of the variable classifying means 132 described above, a limited number of samples were manually handled and verified.

The verification target corresponds to 258 URLs among URLs detected as similar documents, and the numbers and ratios classified by categories according to the verification result by hand are shown in Table 6 below.

category Count ratio sid 72 27.9% referrer 70 27.1% good 36 14.0% bad 41 15.9% Other (if it is difficult to determine the exact meaning of the variable) 39 15.1%

The URLs corresponding to the respective categories among the 258 URLs thus detected are randomly divided into 50 groups, and 50% are used as data for learning, and the remaining 50% are used as data for testing. The accuracy is 99.8% when the category is sid and 99.6% when the category is referrer.

Lastly, the regular expression applying unit 124 according to an embodiment of the present invention deletes a web document which is not meaningful to be redundantly stored as a similar document among the web documents stored in the existing search database 140 or a separate database And / or a specific condition may be generated so that it can be applied to the web crawler 130 so as not to collect similar documents in future web document collection.

First, after the URL clustering and NDD clustering are performed, the regular expression applying unit 124 may generate the regular expression pattern based on the variables classified by the variable classification unit 123 into, for example, the sid category and the referrer category. Here, the regular expression pattern refers to a generalized formula that can be applied to a multiple or an entire URL list in the same URL cluster. It is merely an example to generate a regular expression pattern by removing the variable of the sid or referrer category, and sid and referrer It is also possible to prevent similar document collection by generating a regular expression pattern that can remove variables corresponding to similar documents regardless of whether or not they have an IP address. Therefore, it is easy for a person having ordinary skill in the art to create regular expressions in various ways or expressions besides those described in this specification.

The regular expression applying unit 124 according to an embodiment of the present invention deletes a similar document with respect to web documents stored in an existing search database 140 or a separate database by using the regular expression pattern generated above, May be applied to the web crawler 130 so that the similar document may not be collected when the web document is collected in the future.

Next, a search result providing method according to an embodiment of the present invention will be described with reference to FIG.

According to an embodiment of the present invention, a user may transmit a query term to the search result providing system 100 using his / her user terminal device 300, and the search result providing system 100 may search the search result providing system 100 based on the received query term The user terminal 300 may perform a search by referring to the search database 140 and transmit the result of the search to the user terminal 300. However, in order to prevent duplicate collection of similar documents among the web documents collected by using the web crawler 130, the search result providing system 100 may further include a regular expression < RTI ID = 0.0 > And deletes or deletes the similar document stored in the search database 140 or the like and applies the regular expression generated in the operation of the web crawler 130. FIG. 5 shows each step of this operation as a flowchart As shown in Fig.

5, the search result providing system 100 (or the URL clustering means 121 of the similar document removing unit 120 therein) searches the URLs of the web documents collected by the web crawler 130 For example, URL clustering is performed using a method such as grouping the same clusters if the file names are the same or the paths are the same, and the function of extracting related features by analyzing URLs included in the same cluster (Step S510).

Thereafter, the NDD clustering means 122 of the similar document removal unit 120 performs similarity determination on all pairs of URLs in the same cluster, for example, through "simhash" All similar document pairs are grouped into the same NDD cluster, and the function of extracting features from each NDD cluster based on the bundled NDD clusters can be performed (step S520).

Thereafter, the variable classifying unit 123 of the similar document removing unit 120 classifies each URL cluster, the features extracted from the NDD cluster, and the sample URL based on the similarity between the web document corresponding to the deleted URL and the predetermined variable, It is possible to classify the category of the variable in the URL and determine whether the variable is an unnecessary variable according to the classification (step S530). Here, the category classification operation may be based on a combination of various features and weights for various features obtained through, for example, a machine learning algorithm such as Naive Bayesian classifier or Fisher ' s classifier.

Thereafter, the regular expression applying unit 124 of the similar document removal unit 120 may generate a regular expression pattern for deleting the variable if the variable is determined to be an unnecessary variable (step S540). Thereafter, the generated regular expression pattern is applied to the search database 140 or the like to delete redundant similar documents among the already collected web documents, or to prevent duplicate similar documents from being collected during future web crawling, And it is possible to prevent a large load from being applied to the search.

Embodiments according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as a hard disk, a floppy disk, and a magnetic tape; optical media such as CD-ROM and DVD; magnetic recording media such as a floppy disk; Includes hardware devices specifically configured to store and perform program instructions such as megneto-optical media and ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and specific embodiments and drawings. However, it should be understood that the present invention is not limited to the above- And various modifications and changes may be made thereto by those skilled in the art to which the present invention pertains.

Accordingly, the spirit of the present invention should not be construed as being limited to the embodiments described, and all of the equivalents or equivalents of the claims, as well as the following claims, belong to the scope of the present invention .

Claims (19)

A method for providing a search result corresponding to a search query term of a user,
Clustering a source indication clustering a web document source indication corresponding to a web document stored in a database with a predetermined criterion;
Clustering a similar document in which the search result providing system binds similar documents to the source display clustering result;
Classifying variables using features generated from the source indication clustering step and the similar document clustering step, respectively, by the search result providing system;
Generating a regular expression using the result of the variable classifying step; And
Wherein the search result providing system collects a web document using the generated regular expression,
Wherein the similar document clustering step is performed on the Web document source display lists bundled in the same cluster by the source indication clustering step. The method of claim 1, wherein the web document source indication comprises a URL (Uniform Resource Locator)
Wherein the source indication clustering comprises:
When the search result providing system corresponds to at least one of the cases where the search result providing system is identical to the URLs of the web document from the corresponding URL to the path, the same up to the file name, or the same up to the parameter key, And clustering the search results into clusters.
delete The method according to claim 1,
Wherein the similar document clustering step is performed by the search result providing system, wherein the similarity document clustering step performs similarity determination on all pairs of web document source display in the same cluster. 2. The method of claim 1,
Parsing a web document by the search result providing system;
The hashing of the parsed web document by the search result providing system;
Calculating a simhash by calculating the hashed web document bit by bit according to the search result providing system; And
And grouping the web documents whose hamming distance is less than or equal to a predetermined value as a similar document. The method of claim 1, wherein the feature generated from the source indication clustering comprises at least one of a parameter key related feature and a parameter value related feature. The method of claim 1, wherein the feature generated from the similar document clustering step includes a feature related to a ratio of a unique value of the variable to a total source indication cluster count when creating a similar document pair, A feature associated with a ratio of a unique document participating in creating the search query, a feature associated with the number of unique document versus similar document pairs, and a feature associated with an absolute value of a similar document pair count. The method according to claim 1,
The step of classifying the variables may include classifying categories for each variable included in the web document source indication grouped in one similar document cluster, And classifying the variable as an unnecessary variable when the variable category corresponds to a variable category for checking whether a variable category or a web document is linked. The method according to claim 8, wherein the generating of the regular expression generates the regular expression pattern in which the unnecessary variable is removed from the target web document source display. A system for providing a search result corresponding to a search query term of a user,
A source indication clustering means for grouping, based on a predetermined criterion, web document source indication corresponding to a web document stored in a database;
Similar document clustering means for clustering similar documents with respect to the source indication clustering result;
Variable classifying means for classifying a variable using features generated from the source indication clustering means and the similar document clustering means, respectively; And
A regular expression applying means for generating a regular expression using the result from the variable classifying means; A similar document removing unit including the similar document removing unit,
And a search unit for collecting a web document using the generated regular expression,
Wherein the similar document clustering means performs similar document detection on the Web document source display lists bounded by the same cluster by the source display clustering means. 11. The method of claim 10, wherein the web document source indication comprises a URL (Uniform Resource Locator)
Wherein the source indication clustering means comprises:
The URLs of the web document are grouped into the same cluster when the URLs are the same from the URL to the path, the file names are the same, or the parameter keys are the same. Search result providing system. delete The method of claim 10,
Wherein the similar document clustering means performs similarity determination on all pairs of Web document source display in the same cluster. 11. The apparatus of claim 10,
Parsing a web document;
Hashing the parsed web document;
Computing the hash web document bit by bit to calculate a simhash; And
And grouping the web documents whose hamming distances are equal to or less than a predetermined value as similar documents. 11. The system of claim 10, wherein the feature generated from the source indication clustering means comprises at least one of a parameter key related feature and a parameter value related feature. 11. The method of claim 10, wherein the feature generated from the similar document clustering means comprises a feature relating to a ratio of a unique value of a variable relative to a full source indication cluster count when creating a similar document pair, A feature associated with a ratio of a unique document participating in creating the document, a feature associated with the number of unique document versus similar document pairs, and a feature associated with an absolute value of the similar document pair counter. The method of claim 11,
Wherein the variable classification means classifies categories for each variable contained in the web document source indication grouped into one similar document cluster and classifies the variable category or the web document for managing the user's access session If the search result corresponds to a variable category for checking whether the search result is linked to the search result, the corresponding search result is classified as an unnecessary variable. The search result providing system according to claim 17, wherein the regular expression applying means generates a regular expression pattern for removing the unnecessary variable from the target web document source display. A computer-readable recording medium recording a program for performing each step of the method according to any one of claims 1, 2, and 4 to 9 on a computer.

Images