JP2010123000A - Web page group extraction method, device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To extract Web pages each having similar content in page units as a group by using only URL information from a large number of URLs. <P>SOLUTION: A URL is extracted from an input access log, the URL is regarded as a character string and is divided as a partial character string in each portion, a feature vector is generated based on the appearing partial character string, similarity between the feature vectors is obtained, clustering is performed based on the similarity between the feature vectors, and the URL included in a generated cluster is extracted as the Web page group and is output. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a Web page group extraction method, apparatus, and program, and in particular, when a large number of URLs included in an access log or the like are given, although different URLs are seen, Web pages having similar contents are displayed. The present invention relates to a Web page group extraction method, apparatus, and program for extracting as one group.

  At the time of analyzing the access log, it is analyzed what access is being made to which page based on the URL. At this time, although there are different URLs, there are URLs that should be processed as access to the same Web page or the same type of Web page.

  Examples of different URLs that should be handled as the same Web page include Web pages having the same content such as mirror pages for the purpose of load distribution. Such a page is the same Web page for Web viewers, and even if it is a different URL, the accuracy of analysis is reduced unless it is processed as an access to the same URL.

  In addition, examples of the same type of Web page include a case where a Web page exists for each of many items of the same type.

  For example, a Web page of detailed product information may be prepared individually for many products on an EC (Electronic Commerce) site. Depending on the purpose of log analysis, the accuracy of analysis is improved by processing such a page as an access to a certain type of Web page rather than an access to an individual Web page.

  Because of the above, it is necessary to extract a group consisting of Web pages with the same Web page or the same type of information with different URLs, and make it an analysis target as an access to one Web page.

  It is difficult to manually extract the same Web page or a group of Web pages of the same type from the URL included in the access log.

On the other hand, a method for extracting a Web site having the same contents includes a method for finding a mirror site group. In this method, a page to be a top page of a website is estimated from a large number of web page sets, a site set is determined from a top page estimated for the web page set and pages linked to the top page, and the size of the site set is determined. Sites with a certain value or more are narrowed down as processing targets, and files of link character strings, anchor character strings, and internal / external link information possessed by the sites are created. A site pair having the same characteristics is selected as a mirror site from these, and the mirror site pair is detected from the similarity of the mirror site candidate pairs (see, for example, Patent Document 1).
JP 2004-264926 A

  However, the same Web site extraction method described above has the following problems.

  (1) The same Web site extraction method uses the URL included in the access log and the link structure included in the Web page indicated by the URL to find the same site. For this reason, when analyzing an access log, it is not always possible to acquire a Web page at the time of accessing an included URL, and there are many cases where information included in the Web page cannot be used. In such a case, there is a problem that overlapping Web pages cannot be extracted.

  (2) The above extraction method aims to find the same site in units of sites. For this reason, Web pages having the same contents are not necessarily included in the mirror site of the entire site, and duplicate pages are often prepared only for specific pages where access is concentrated on a certain site. In such a case, the entire site is not necessarily duplicated, and there is a problem that duplicate web pages cannot be extracted.

  (3) In addition, the above extraction method extracts the completely same Web page, and Web pages that are considered to be access to the same type of information cannot be extracted.

  As described above, the conventional method has a problem that a Web page must be acquired in the access log analysis, and is an extraction method in units of sites.

  The present invention has been made in view of the above points, and a Web page group extraction method capable of extracting, as a group, Web pages having similar contents in units of pages using only URL information from a large number of URLs. An object is to provide an apparatus and a program.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) is a method for extracting the same Web page or the same type of Web page as a group from the access log, which are different URLs.
A URL extraction step (step 1) in which the access log input means extracts a URL from the input access log and stores it in the URL storage means;
A character string dividing step (Step 2) in which the character string dividing means reads the URL from the URL storage means, regards the URL as a character string, and divides the URL as a partial character string for each part;
A feature vector calculation means for generating a feature vector based on the appearing partial character string and storing it in the feature vector storage means (step 3);
A similarity calculation unit reads out a feature vector from the feature vector storage unit, obtains a similarity between the feature vectors, and stores it in the similarity storage unit (step 4);
The clustering means reads out the similarity between the feature vectors from the similarity storage means, performs clustering, extracts the URL included in the generated cluster as a Web page group, and outputs it to the URL classification storage means (step 5) ) And do.

Further, the present invention (Claim 2) is a character string dividing step (Step 2).
Divided into host part, domain part, directory part, query part as partial character string of URL,
In the feature vector calculation step (step 3),
A feature vector is obtained based on the appearance frequency of the partial character string.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 3) is a Web page group extracting apparatus that extracts the same Web page or the same type of Web page as a group from the access log, which are different URLs.
URL extraction means 2 for extracting a URL from the input access log and storing it in the URL storage means 3;
A character string dividing unit 41 that reads a URL from the URL storage unit 3, regards the URL as a character string, and divides the URL as a partial character string for each part;
A feature vector calculation means 42 for generating a feature vector based on the appearing partial character string and storing it in the feature vector storage means 5;
A similarity calculation unit 43 that reads out a feature vector from the feature vector storage unit 5, calculates a similarity between the feature vectors, and stores it in the similarity storage unit 6;
Clustering means 44 that reads out the similarity between feature vectors from the similarity storage means 6 and performs clustering, extracts URLs included in the generated cluster as a Web page group, and outputs them to the URL classification storage means 7; .

Further, according to the present invention (claim 4), in the character string dividing means 41,
Including means for dividing the URL part character string into a host part, a domain part, a directory part, and a query part,
The feature vector calculation means 42
Means for obtaining a feature vector based on the appearance frequency of the partial character string is included.

  The present invention (Claim 5) is a Web page group extraction program for causing a computer to function as each means constituting the Web page group extraction apparatus according to any one of Claims 3 and 4.

  As described above, according to the present invention, only URL information is used from a large number of URLs, and the same Web page or / and the same type of Web page having different URLs can be extracted as a group. By extracting a group of Web pages, access to those Web pages can be handled as access to the same Web page, and an improvement in access log analysis accuracy can be expected.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 3 shows the configuration of the URL classification device in one embodiment of the present invention.

  The URL classification apparatus shown in FIG. 1 includes an access log input unit 2, a URL storage unit 3, a URL classification unit 4, a feature vector storage unit 5, a similarity storage unit 6, and a URL classification storage unit 7.

  When the collected access log 1 is input, the access log input unit 2 extracts a URL from the access log and stores it in the URL storage unit 3.

  The URL classifying unit 4 includes a character string dividing unit 41, a feature vector calculating unit 42, a similarity calculating unit 43, and a clustering unit 44. The character string dividing unit 41 reads the URL from the RL storage unit 3 and divides the character string included in the URL for each part, and the feature vector calculating unit 42 counts the partial character strings to obtain the URL feature vector. The similarity calculation unit 43 calculates the similarity between URL feature vectors, the clustering unit 44 performs clustering, and stores the result in the URL classification storage unit 7.

  The operation in the above configuration will be described below.

  FIG. 4 is a flowchart of the operation of the URL classification unit according to the embodiment of the present invention.

  Step 101) First, the character string dividing unit 41 reads out the access log from the URL storage unit 3, and for each URL included in the access log as shown in FIG. Further, the directory part is extracted for each layer, and the query part is extracted for each parameter.

  Examples of division are shown below.

URL example:
http: //www.xxx.yuu/path1/path2/path3? param1 = value1 & param2 = value2
Example of division:
host part [www]
domain part [xxx.yyy]
directory part [path1, path2, path3]
query part [param1 = value1, param2 = value2]
Step 102) The feature vector calculation unit 42 counts the partial character strings included in the URL, obtains a URL feature vector based on the partial character string, and stores it in the feature vector storage unit 5. At this time, partial character strings included in different parts are treated as different partial character strings.

  Note that the number of vector components is the number p of all partial character strings occurring in the host part of all target URLs, and the number q of all partial character strings occurring in each domain part, path part, and query part. The sum of r and s is (p + r + s = N) N.

  FIG. 7 shows a part of the URL feature vector calculated when the 13 URL examples shown in FIG. 6 are given.

In the URL example of FIG. 6, the number of types of partial character strings is
host part: 4, domain part: 5, directory part: 11, query part: 5
And is represented by a feature vector of 25 dimensions.

  The numbers in FIGS. 7 to 9 indicate URLs having the same row numbers in FIG.

  Step 103) Next, the similarity calculation unit 43 reads the URL feature vector from the feature vector storage unit 5, calculates the similarity between the URL feature vectors, and stores it in the similarity storage unit 6.

  There are several methods for calculating the similarity sim (a, b) between two URL feature vectors in the similarity calculation unit 43. For example, an inner product between two URL feature vectors can be used. FIG. 8 shows two URL similarities in each of the URL lists of FIG. The similarity storage unit 6 stores the distance between URL feature vectors shown in FIG.

  Step 104) The clustering unit 44 reads the similarity (distance between URL feature vectors) from the similarity storage unit 6 and collects similar URL feature vectors into the same cluster. Several methods are conceivable as methods for grouping into clusters. For example, the shortest distance method can be used (reference: Kazuaki Kishida, “Document clustering technique: literature review”, Library and Information Science, no.49, pp.33-75 (2003)). A vector having a higher degree of similarity is considered to be a vector located closer.

  The clustering method using the shortest distance method is shown below.

  1) All URL feature vectors are individually set as initial clusters, and processing is started.

  2) Recalculate the distance between the combined cluster and other clusters, using the distance between the closest objects among the objects belonging to the two clusters as the intercluster similarity.

  The above steps 1) and 2) are repeated until there is no cluster having a distance equal to or smaller than a preset threshold value.

  FIG. 9 shows a tree diagram of the clustering result of the URL list of FIG. URL groups having URL feature vectors included in the same cluster are extracted as web pages having the same contents and stored in the URL classification storage unit 7.

In FIG. 9, if a threshold is set at the position indicated by the line,
URL numbers 1 and 2;
URL numbers 9, 10 and 11;
URL numbers 3, 4 and 5;
URL numbers 6, 7 and 8;
URL number 12;
URL number 13;
7 clusters are obtained. This is the same as the expected URL group.

  By performing the processing as described above, only URL information is used from a large number of URLs included in the access log (not related to the contents of the URL) and has different URLs, but the same Web page or / and the same Web pages of types can be extracted as a group.

  It is possible to construct the operation of the components of the apparatus shown in FIG. 3 as a program, install it on a computer used as a Web page group extraction apparatus, and execute it, or distribute it via a network. .

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention can be applied to a technique for searching a Web page.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the Web page group extraction apparatus in one embodiment of this invention. It is a flowchart which shows the detailed process of the URL classification | category part in one embodiment of this invention. It is an example of the URL character string in one embodiment of this invention. It is a specific example of a URL list in an embodiment of the present invention. It is a specific example of the URL feature vector in one embodiment of the present invention. It is a specific example of the distance between URL feature vectors in an embodiment of the present invention. It is an example of the clustering result in one embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Access log 2 URL extraction means, Access log input part 3 URL memory | storage means, URL memory | storage part 4 URL classification | category part 5 Feature vector memory | storage means 6 Similarity storage means 7 URL classification | category storage means 41 Character string division | segmentation means, Character string division | segmentation part 42 Feature vector calculation means, feature vector calculation section 43 similarity calculation means, similarity calculation section 44 clustering means, clustering section

Claims (5)

A method for extracting the same Web page or the same type of Web page as a group from an access log with different URLs,
A URL extraction step in which the access log input means extracts a URL from the inputted access log and stores it in the URL storage means;
A character string dividing unit that reads the URL from the URL storage unit, regards the URL as a character string, and divides the URL as a partial character string for each part; and
A feature vector calculating means for generating a feature vector based on the appearing partial character string and storing it in the feature vector storage means;
A similarity calculation unit reads out a feature vector from the feature vector storage unit, obtains a similarity between the feature vectors, and stores the similarity in the similarity storage unit;
A clustering unit that reads out the similarity between feature vectors from the similarity storage unit, performs clustering, extracts a URL included in the generated cluster as a Web page group, and outputs the URL to the URL classification storage unit;
A Web page group extraction method characterized by: In the string splitting step,
Divided into the host part, domain part, directory part, query part as a partial character string of the URL,
In the feature vector calculation step,
The Web page group extraction method according to claim 1, wherein the feature vector is obtained based on the appearance frequency of the partial character string. A Web page group extraction device that extracts the same Web page or the same type of Web page as a group from an access log with different URLs,
URL extraction means for extracting a URL from the input access log and storing it in the URL storage means;
Character string dividing means for reading the URL from the URL storage means, regarding the URL as a character string, and dividing the URL as a partial character string for each part;
A feature vector calculating means for generating a feature vector based on the appearing partial character string and storing it in a feature vector storage means;
A similarity calculation unit that reads out a feature vector from the feature vector storage unit, obtains a similarity between the feature vectors, and stores the similarity in the similarity storage unit;
Clustering means for reading out similarity between feature vectors from the similarity storage means, performing clustering, extracting URLs included in the generated cluster as Web page groups, and outputting them to URL classification storage means;
A Web page group extracting apparatus characterized by comprising: The character string dividing means is
Means for dividing the URL partial character string into a host part, a domain part, a directory part, and a query part;
The feature vector calculation means includes:
The Web page group extraction device according to claim 3, further comprising means for obtaining the feature vector based on the appearance frequency of the partial character string.   A Web page group extraction program for causing a computer to function as each means constituting the Web page group extraction device according to claim 3.

Images