JP5135174B2 - Large-scale WEB site evaluation apparatus, large-scale WEB site evaluation method, and large-scale WEB site evaluation program - Google Patents

Description

  The present invention evaluates in advance whether or not an important site for a WEB service having a large number of electronic documents, so that when searching for necessary electronic documents using keywords or the like from these electronic documents, The present invention relates to a technique that is useful when priority is given to highly evaluated electronic documents.

  More specifically, a URL indicating an acquisition source of an electronic document acquired in advance, related information (referred to as a link) with other documents described in the document, and a group of documents as a group of WEB services. Using a database that is classified into sites based on URLs for handling, it is possible to achieve document search with higher accuracy than before by sorting documents into sites and evaluating each site. It is related with the evaluation apparatus, method, and program of a large-scale WEB site.

  Conventionally, when a full-text search is performed on a WEB document, there is a page rank as one of the determination criteria for rearranging the search results according to the priority order. This is an evaluation criterion for giving priority to a page with a large number of points indicated by links, which is considered to be related from a larger number of pages.

However, in recent years, even though there is not much relevance, this evaluation is improved by creating a lot of links to your own page where you want to get a high evaluation value from the many pages you created yourself. The act of doing that is increasing. When such a link is used as it is, the search result often becomes unfavorable. These are called link spam, and there is a technique for detecting a WEB server that would constitute an invalid link from the page rank of the WEB server unit (host) as disclosed in Non-Patent Document 1. .
Gyongyi, Z .; , Garcia-Molina, H .; and Pedersen, J. et al. , Link Spam Detection Based on Mass Estimation, VLDB '06: Proceedings of the 32nd international conference on Very large data bases, 2006. Hiroki Takada (Nara Institute of Science and Technology), Takeshi Yamada, Nobuyoshi Ueda (NTT Communication Science Laboratories), "Clustering Based on Functional Characteristics of Nodes", Proceedings of Network Ecology 2008 Symposium, pp. 120-124, 2008

  In a blog or the like, a plurality of users individually create a WEB document on the same host, and if the users are handled in units of hosts as in the detection technique described in Non-Patent Document 1, all users are handled together. It will be.

  In the present invention, the above-mentioned problems are solved, and a plurality of WEB documents acquired from a specific location set in advance for each server or a WEB document are collected in units of sites and described in the WEB document. By linking and analyzing link information in units of sites, an invalid self-created link by the page creator is handled, or an evaluation value equivalent to the page rank is given to each WEB document by treating it with a low evaluation, and at the time of search Make effective use of priorities.

  In addition, when analyzing link information, as disclosed in Non-Patent Document 2, the link structure is analyzed between the three sites, thereby efficiently calculating the evaluation value and the relationship between the sites. A large-scale WEB site evaluation apparatus, a large-scale WEB site evaluation method, and a large-scale WEB site evaluation program are provided.

  In the present invention, for a WEB document group acquired from the WEB, information on the WEB document is aggregated and handled based on the site aggregation rule, analyzed by the site analyzer, and a new site aggregation rule is added. In addition, it is configured such that a highly accurate evaluation value of a WEB document can be assigned at high speed by accumulating information with an appropriate evaluation value based on the site aggregation rule in an evaluated WEB document storage device.

  That is, the evaluation apparatus for a large-scale WEB site according to claim 1 includes a WEB document storage unit that stores information about a WEB document acquired from the WEB, and a site aggregation rule in which site aggregation rules for aggregating the sites are accumulated. Based on the site aggregation rules stored in the storage means and the site aggregation rule storage means, link information of the WEB documents stored in the WEB document storage means is aggregated on a site basis, and the links between the aggregated sites Analyzing the structure, obtaining an evaluation value corresponding to the page rank for each of the aggregated sites, and based on the analyzed link structure, the obtained evaluation value for each site and the link information, To create a new site aggregation rule and update the site aggregation rule stored in the site aggregation rule storage means Site analysis means, and for each WEB document stored in the WEB document storage means, for each document using link information between sites based on the site aggregation rules stored in the site aggregation rule storage means. WEB document evaluation means for assigning an evaluation value; and evaluated WEB document storage means for storing a WEB document assigned an evaluation value by the WEB document evaluation means.

  Further, the evaluation apparatus for a large-scale WEB site according to claim 2 is characterized in that, in claim 1, the site analysis unit stores the WEB document storage unit based on the site aggregation rule stored in the site aggregation rule storage unit. Site aggregation means for aggregating the link information of the stored WEB documents for each site, link structure analysis means for analyzing the link structure between sites aggregated by the site aggregation means, and sites aggregated by the site aggregation means And link information, a site evaluation means for obtaining an evaluation value corresponding to a page rank for each site, a link structure analyzed by the link structure analysis means, an evaluation value for each site obtained by the site evaluation means, and the Based on the link information, create a new site aggregation rule by grouping highly related sites together, It is characterized and site determination means for updating the stored sites aggregated rule site aggregation rule storage means, further comprising: a.

  Further, the evaluation apparatus for a large-scale WEB site according to claim 3, according to claim 1, wherein the site aggregation rule accumulating unit divides the host name into a host name and a directory name of the acquisition source information of the WEB document. It is characterized by the accumulation of structured site aggregation rules that are handled individually and grouped common parts.

  According to a fourth aspect of the present invention, there is provided a method for evaluating a large-scale WEB site, wherein a WEB document acquisition unit acquires a WEB document from the Internet and stores information related to the WEB document in the WEB document storage unit; A site aggregating step for aggregating the link information of the WEB document accumulated in the WEB document accumulating unit for each site based on the site aggregation rule for aggregating the sites accumulated in the site aggregation rule accumulating unit; A link structure analysis step in which the link structure analysis means analyzes the link structure between the sites aggregated by the site aggregation means; and a site evaluation means acquires the site and link information aggregated by the site aggregation means; The site evaluation step for obtaining an evaluation value equivalent to the page rank every time and the site discrimination means Based on the link structure analyzed by the link structure analysis means and the evaluation value for each site obtained by the site evaluation means and the link information, a site aggregation rule is newly created by collecting strongly related sites, A site determination step for updating the site aggregation rule stored in the site aggregation rule storage unit, and a WEB document evaluation unit stores in the site aggregation rule storage unit for each WEB document stored in the WEB document storage unit And a step of assigning an evaluation value for each document using link information between sites based on the site aggregation rule and storing the evaluation value in the evaluated WEB document storage means.

  An evaluation program for a large-scale WEB site according to claim 5 is an evaluation program for a large-scale WEB site that causes a computer to function as each means according to any one of claims 1 to 3.

(1) According to the first to fifth aspects of the present invention, a page rank evaluation value is given by aggregating WEB documents in units of sites and creating a new site aggregation rule from the page rank evaluation and link structure. Therefore, the links to be used can be effectively selected and calculated at high speed, and it is possible to realize a document search with higher accuracy than before.
(2) According to the invention described in claim 3, the site can be aggregated at high speed by using the structured site aggregation rule.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments.

  As shown in FIG. 1, the evaluation apparatus for a large-scale WEB site according to the present embodiment stores a WEB document in which an existing WEB document acquisition apparatus 100 stores a WEB document acquired from a WEB server via the Internet. Based on the information in the WEB document storage device 200 as a means, using the site aggregation (classification) rules in the site aggregation rule database 400 as the site aggregation rule storage means, the aggregation (classification) to the site and the analysis result A site analysis apparatus 300 serving as a site analysis unit for updating the site aggregation rule, and a WEB for assigning an evaluation value to each of the accumulated WEB documents in the WEB document accumulation apparatus 200 based on the site aggregation rule in the database 400 WEB document evaluation apparatus 500 as document evaluation means, and WEB document evaluation apparatus 500 It is constructed by the evaluated configuration WEB document storage device 600 as the evaluated WEB document storage means for storing the WEB document more evaluation value is assigned.

  Each function described below of the WEB document acquisition apparatus 100, the site analysis apparatus 300, and the WEB document evaluation apparatus 500 is achieved by, for example, a computer.

  The site analysis apparatus 300 is configured as shown in FIG. In FIG. 2, reference numeral 301 denotes a site aggregating apparatus as a site aggregating means for aggregating WEB document link information accumulated in the WEB document accumulating apparatus 200 on a site basis based on an arbitrary site aggregation rule.

  Reference numeral 302 denotes an aggregate data storage device that accumulates information aggregated for each site. Reference numeral 303 denotes a link structure analysis device as a link structure analysis unit that acquires a set of sites and link information from the aggregate data storage device 302 and analyzes the link structure between sites.

  Reference numeral 304 denotes a link structure analysis auxiliary recording device in which information obtained by combining link information of a link destination site and a site adjacent to the link source site is recorded by the link structure analysis device 303.

  Reference numeral 305 denotes a site evaluation device serving as a site evaluation unit that acquires aggregated site and link information from the aggregate data storage device 302 and evaluates the site.

  Reference numeral 306 denotes a site evaluation auxiliary recording device that records site evaluation values distributed for each site by the site evaluation device 305.

  Reference numeral 307 denotes an analysis data storage device that stores the results obtained from the link structure analysis device 303 and the site evaluation device 305.

  Reference numeral 308 denotes a site discriminating device as site discriminating means for creating a rule for acquiring information from the analysis data storage device 307 and newly aggregating it as a site.

  As shown in FIG. 3, the site aggregation rule in the site aggregation rule database 400 corresponds to the host name and directory name of the WEB document acquisition source information (URL, that is, http: // host name / directory name). When the aggregation rule is met, the described aggregation site name is given. Basically, a WEB document collects host names as headings (site names), and describes a user site (in this example, blog.goo.ne.jp/?) Where the host name and directory name match. Rule) is aggregated with the user site name as a heading (site name). A user site is a unit that collects a group of WEB documents that can be freely created by a service user, in which a part of a URL such as a homepage service or a blog of a provider that provides an Internet service has a certain regularity.

The aggregation site name is the name of the temporary order to summarize the multiple hosts, for example, as shown in FIG. 3, ^*. ntt. co. jp ^* . ntt. jp and a rule are described, a WEB document having a host name “arbitrary word.ntt.co.jp” or a WEB document having “arbitrary word.ntt.jp” is represented by ntt. co. By assigning an aggregate site name of jp, it means that the host servers belong to the same site.

  There may be a plurality of site aggregation rules with respect to one aggregation site name as in the example of FIG. The site aggregation rule is created in advance for a known blog service or the like.

  Further, if the site aggregation rule is used as it is, it is necessary to perform a conformity inspection according to the number of rules for each page, which takes processing time. In the present invention, the site aggregation rule is divided into a host name and a directory name. By constructing and using the structured site aggregation rule as shown in FIG. 4 in which common parts are individually handled in ‘hierarchy’ and hierarchized, high-speed aggregation is realized by performing the conformity inspection in a small number of times.

  The WEB document evaluation apparatus 500 reads out a WEB document from the WEB document storage apparatus 200 and assigns an evaluation value for each document from link information between the WEB documents based on the structured site aggregation rule in the site aggregation rule database 400. At this time, the link information used for evaluation deals only with link information between different sites. This is because links within the same site can be freely created by those managing the site, and thus cannot be an objective index for evaluating a page. By this mechanism, it is possible to eliminate the unrelated links, which was a problem of the conventional method, and to calculate the evaluation value using the effective links.

  The evaluated WEB document storage device 600 stores together the evaluation values given by the WEB document evaluation device 500, and stores the WEB document in a format convenient for use in full-text search or the like.

  Next, details of the site analysis apparatus 300 will be described. The site aggregation device 301 acquires the URL of the WEB document and the link information described in the WEB document as a set from the WEB document storage device 200 in which data as shown in FIG. Based on the structured site aggregation rule, the host name or user site name corresponding to each URL is found (site name) only when the URL of the WEB document and the link destination URL do not have the same aggregation site name. As well as the link information.

  At this time, as shown in FIG. 6, the link destination site is first added to the site of the WEB document, and further, the addition of the link source site to the link destination site is repeatedly recorded. Thus, when all the WEB documents recorded in the WEB document storage apparatus 200 are processed, the link destination and link source site information are aligned for each site.

  The link structure analysis device 303 acquires the site and link information as a set from the aggregate data storage device 302, and based on the link information, combines the link information of the link destination site and the site adjacent to the link source site. The data is recorded in the link structure analysis auxiliary recording device 304. If all the data recorded in the aggregated data storage device 302 is processed, link information between a site and its adjacent sites is collected as shown in FIG. Next, the link structure analysis device 303 sequentially analyzes the data recorded in the link structure analysis auxiliary recording device 304 for the three-site link structure for each site, and uses the pattern as data representing the characteristics of the link structure of the site. The data is recorded in the analysis data storage device 307 for each site.

  First, the site evaluation device 305 acquires the site and link information as a set from the aggregate data storage device 302, distributes the evaluation value of the site to the link destination site based on the page rank method, and assists in site evaluation. Recording is performed on the recording device 306.

  Next, the site evaluation device 305 aggregates the evaluation values distributed for each site from the site evaluation auxiliary recording device 306, updates them as new evaluation values, and calculates the difference between the new evaluation value and the previous evaluation value. The maximum difference is recorded in the entire site evaluation auxiliary recording device 306. When this maximum difference is equal to or greater than a predetermined value, the updated new evaluation value is distributed to the linked sites and aggregated to update the evaluation value.

  At this time, depending on the site, there is a link destination site, but when there is no link source site, the evaluation value of the site is determined to be a constant value. In the case of such a site, the site evaluation device 305 marks the fixed value, distributes the evaluation value to the link destination site, and records the evaluation value and the link information to the analysis data storage device 307 so that the site evaluation device 305 If there is a next time to perform a calculation, there is no need for unnecessary calculations. Further, even if there is a link source site, the processing amount can be reduced by processing in the same manner even when all of the link source sites become fixed values. When the maximum difference falls below a predetermined arbitrary value, the site evaluation value and link information of the site evaluation auxiliary recording device 306 are recorded in the analysis data storage device 307, and the process ends.

  The site discriminating apparatus 308 uses the discriminant from the evaluation value, the link information, and the characteristics of the link structure for each site, and if the site names can be inferred to be highly relevant even if there is no common point, Combine multiple site aggregation rules into a new site.

Example 1
Examples of the present invention will be described below.

  The existing WEB document acquisition apparatus 100 acquires a WEB document from the Internet, and records the acquisition source URL, information about the WEB document, and link information in the WEB document in the WEB document storage apparatus 200. The link information in the WEB document is a basic function for efficiently acquiring WEB documents one after another by following links in the existing WEB document acquisition apparatus 100, so that this information can also be recorded. good. Here, it is assumed that data as shown in FIG. 5 has been acquired.

  An example of the site aggregation rule in the site aggregation rule database 400 is shown in FIG. Aggregation site name example. jp is ex1. example. jp and ex2. example. For a WEB document having a host name such as jp, an aggregate site name example. It is a rule that jp is assigned. The next level rule in FIG. co. jp and ntt. jp has a different domain name, but the aggregate site name is ntt. co. jp is treated as the same site. Further, the rule in the next stage in FIG. 3 is that, even if the host name portion is the same, but the creator is different for each user depending on the directory name portion, for example, an ISP (Internet Service Provider) user homepage service or a blog This is a rule for handling services and the like by author, and the aggregation site name means that it is treated as a user site name as a site name at the time of aggregation.

  When used, the site aggregation rule is structured as shown in FIG. 4 as a structured site aggregation rule. In the site aggregation device 301 and the web document evaluation device 500 of the site analysis device 300, the URL (http: // host name / directory name) acquired from the web document storage device 200 is divided into a host name and a directory name, and the host name Furthermore, '. Divide by 'and handle in reverse order from the right. For example, URL www. ntt. co. jp / top. html is the host name www. ntt. co. jp and directory name / top. The host name is further divided into html, jp, co, ntt, and www. When this is compared with the structured site aggregation rule of FIG. 4 from the top, it matches [jp] + [co] + [ntt] + [arbitrary], and the aggregation site name ntt. co. jp is assigned. Similarly, the link information www. ntt. jp / is an aggregate site name ntt. co. jp, www. ntt-east. Since there is no matching rule, jp / ntt-east. jp is the site name.

  The site aggregation device 301 does not record the aggregation data storage device 302 when the aggregation site name is the same. In this case, the site name www. ntt. co. jp as the link destination, the site name www. ntt-east. jp and the site name www. ntt-east. jp, the site name www. ntt. co. jp is recorded. FIG. 6 illustrates the result of processing the recorded data example of FIG. 5 and recording it in the aggregated data storage device 302 by the same procedure.

  The link structure analysis device 303 acquires the site name and link information from the aggregate data storage device 302, and links the site name and the processed link information for each link destination and link source site to the link structure analysis auxiliary recording device 304. To record. In the example of FIG. jp is blog. as link information and blog. goo. ne. jp / person1, blog. goo. ne. I have jp / person2. Since the link information is processed in order, first, the site blog. goo. ne. jp / person1 is linked to example. jp and link information other than the link information with the recording destination site is added as the link information. Here, (first: none, original: blog.goo.ne.jp/person2) is recorded together. Next, the site blog. goo. ne. jp / person2 is linked to example. jp and (link: good.ne.jp/person1, source :) are recorded together as link information. The result of processing and recording the data illustrated in FIG. 6 is an example of the link structure analysis auxiliary recording device 304 of FIG.

  Next, the link structure analysis device 303 considers this link information as a set of three sites for each site, and analyzes the connection method. example. In the case of jp, a combination of 3 sites can be combined with the link information site. In addition, the relationship between the direct link destination and the link source can be analyzed from the added link information, and it can be seen that this has a link structure as shown in FIG. By assigning the numbers shown in FIG. 10 to the roles in this link structure, the characteristics of what role the site has are expressed. That is, the link structure has 13 patterns, and the role of the site is classified into 30 types. This feature is recorded in the analysis data storage device 307 as a site feature.

  The site evaluation device 305 acquires the site name and link information from the aggregate data storage device 302 and records the evaluation value at the link destination. For example, the site www. ntt. co. In the case of jp, the site www. ntt. co. jp and the initial value 1.0 of the evaluation value, information on the link destination site is recorded, and the link destination site www. ntt-east. The value obtained by multiplying jp by the initial value 1.0 multiplied by the number of links to the linked site (here, 1) and divided by the total number of linked sites (here, 1) is recorded. FIG. 8 illustrates the result of processing the entire data of the aggregate data storage device 302 by the site evaluation device 305 (recorded data in the site evaluation auxiliary recording device 306).

  Next, the site evaluation apparatus 305 calculates an evaluation value for each site using the following formula (1). Arbitrary values can be specified for the initial value and weight. In this example, the calculation is performed with an initial value of 1.0 and a weight of 0.85.

Evaluation value = initial value + weight × total sum of link source evaluation values (1)
The site evaluation device 305 records the absolute value of the difference between the calculated value and the previous evaluation value, and repeatedly performs the calculation until it becomes lower than a predetermined threshold value. In the next calculation, the newly calculated evaluation value is multiplied by the number of links to the linked site, and the value divided by the total number of linked sites is recorded in the linked site. The basic calculation method is as described above. Here, the site evaluation device 305 sets the final evaluation value (site rank) in the case of the initial value 1.0, the initial value 1.0 only for a specific site, and others. Site 2 calculates the evaluation value (credit rating rank) calculated with the initial value 0.0. The latter specific site designates a site that is considered to be linked only to a site that does not create an unrelated link. For example, a governmental site (.go.jp) or an academic site ( ^* .ac.jp). Here, ntt. co. The trustworthiness rank is calculated with only the initial value 1.0 for jp. A value obtained by evaluating these two evaluation values by the following expression (2) is recorded in the analysis data storage device 307 as a final site evaluation value.

Site evaluation value = 1.0−credit rating rank / site rank (2)
The site discriminating apparatus 308 compiles sites that are strongly related to each other by using a predetermined discriminating rule or a site discriminator based on machine learning based on link information, link structure characteristics, and site evaluation values for each site. Created and recorded in the site aggregation rule database 400. In this example, from the recorded data shown in FIG. goo. ne. jp / person1 and blog. goo. ne. As a new site aggregation rule with jp / person2 as the same site, the rule shown in FIG. 12 is added.

  In addition to the site aggregation rule added as described above, the WEB document evaluation apparatus 500 sets an evaluation value for each WEB document with an initial value of 0.15 and a weight of 0.85 in the same manner as the site evaluation apparatus 305. Calculate. When the absolute value of the difference between the evaluation values is lower than a certain threshold value, or a result calculated for a certain number of times is added and recorded in the evaluated WEB document storage device 600. Based on this data, it will be applied to full-text search services.

  The embodiment of the evaluation method for a large-scale WEB site according to the present invention executes the processing of each device described with reference to FIGS. That is, the step in which the WEB document acquisition apparatus 100 acquires a WEB document from the Internet and stores information on the WEB document in the WEB document storage apparatus 200, and the site aggregation apparatus 301 of the site analysis apparatus 300 includes the site aggregation rule database 400. Based on the site aggregation rules for aggregating the sites, the site aggregation step for aggregating the link information of the WEB documents stored in the WEB document storage device 200 in units of sites, and for each site A link structure analysis device 303 acquires a site and link information as a set from the aggregate data storage device 302, and a link structure analysis auxiliary recording device for the link structure between sites. Link structure analysis step for analyzing using 304 A site evaluation step in which the site evaluation device 305 acquires the aggregated site and link information from the aggregate data storage device 302 and obtains an evaluation value corresponding to the page rank for each site while using the site evaluation auxiliary recording device 306; The step of accumulating the results obtained by the link structure analysis device 303 and the site evaluation device 305 in the analysis data storage device 307, and the site determination device 308 obtain information from the analysis data storage device 307, so A site determination step for collectively creating a new site aggregation rule and updating the site aggregation rule stored in the site aggregation rule database 400, and a WEB document evaluation device 500, each of which is stored in the WEB document storage device 200 For the WEB document, the site aggregation rule database is used. 400 based on the accumulated site aggregation rules, granted an evaluation value for each document using the link information between sites, and a step of storing the evaluated configuration WEB document storage device 600.

  Further, a part or all of the functions of each means in the evaluation apparatus for a large-scale WEB site of the present embodiment can be configured by a computer program, and the present invention can be realized by executing the program using the computer. It goes without saying that the procedure in the evaluation method for a large-scale WEB site of the present embodiment can be constituted by a computer program, and the program can be executed by the computer. Readable recording media such as FD (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk) -ROM, DVD (Dig) It is possible to record and save or distribute it in (Versatile Disk) -ROM, CD-R, CD-RW, HDD, removable disk or the like. It is also possible to provide the above program through a network such as the Internet or electronic mail.

The block diagram which shows the example of embodiment of the evaluation apparatus of the large-scale WEB site of this invention. The block diagram which shows the principal part structure of the apparatus of FIG. Explanatory drawing which shows an example of the site aggregation rule in the embodiment of this invention. Explanatory drawing which shows an example of the structured site aggregation rule in the embodiment of this invention. Explanatory drawing which shows an example of the recording data in the WEB document storage apparatus in the embodiment of this invention. Explanatory drawing which shows an example of the recording data in the aggregated data storage apparatus in the embodiment of this invention. Explanatory drawing which shows an example of the recording data in the link structure analysis auxiliary | assistant recording device in the embodiment of this invention. Explanatory drawing which shows an example of the recording data in the site evaluation auxiliary | assistant recording device in the example of embodiment of this invention. Explanatory drawing which shows an example of the link structure analyzed with the link structure analyzer in the embodiment of this invention. Explanatory drawing showing the feature number of the link structure in the embodiment of this invention. Explanatory drawing which shows an example of the recording data in the analysis data storage device in the embodiment of the present invention. Explanatory drawing which shows an example of the site aggregation rule added to the site aggregation rule database in the embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... WEB document acquisition apparatus, 200 ... WEB document storage apparatus, 300 ... Site analysis apparatus, 301 ... Site aggregation apparatus, 302 ... Aggregation data storage apparatus, 303 ... Link structure analysis apparatus, 304 ... Link structure analysis auxiliary recording apparatus, 305 ... site evaluation apparatus, 306 ... site evaluation auxiliary recording apparatus, 307 ... analysis data storage apparatus, 308 ... site discrimination apparatus, 400 ... site aggregation rule database, 500 ... WEB document evaluation apparatus, 600 ... evaluated WEB document storage apparatus.

Claims (5)

WEB document storage means for storing information about a WEB document acquired from WEB;
Site aggregation rule storage means for storing site aggregation rules for aggregating sites,
Based on the site aggregation rule stored in the site aggregation rule storage unit, the link information of the WEB document stored in the WEB document storage unit is aggregated for each site, and the link structure between the aggregated sites is analyzed. , An evaluation value corresponding to the page rank is obtained for each of the aggregated sites, and based on the analyzed link structure, the obtained evaluation value for each site, and the link information, the strongly related sites are collectively updated. A site analysis unit that creates a site aggregation rule and updates the site aggregation rule stored in the site aggregation rule storage unit;
WEB for assigning an evaluation value for each document using link information between sites based on the site aggregation rule stored in the site aggregation rule storage unit for each WEB document stored in the WEB document storage unit Document evaluation means;
An evaluated WEB document storage means for storing the WEB document to which the evaluation value is given by the WEB document evaluation means;
An evaluation apparatus for a large-scale WEB site characterized by comprising: The site analysis means includes
A site aggregating unit for aggregating link information of WEB documents accumulated in the WEB document accumulating unit for each site based on the site aggregation rules accumulated in the site aggregation rule accumulating unit;
A link structure analysis means for analyzing a link structure between sites aggregated by the site aggregation means;
Site evaluation means for obtaining site and link information aggregated by the site aggregation means, and obtaining an evaluation value equivalent to page rank for each site;
Based on the link structure analyzed by the link structure analysis unit and the evaluation value for each site obtained by the site evaluation unit and the link information, a site aggregation rule is newly created by combining strongly related sites, Site discrimination means for updating the site aggregation rules stored in the site aggregation rule storage means;
The apparatus for evaluating a large-scale WEB site according to claim 1, comprising:   The site aggregation rule accumulating means accumulates structured site aggregation rules that are divided into a host name and a directory name of the acquisition source information of the WEB document, and the host names are handled individually and the common parts are hierarchized. The evaluation apparatus for a large-scale WEB site according to claim 1 or 2, characterized in that: WEB document acquisition means acquires a WEB document from the Internet, and stores information about the WEB document in the WEB document storage means;
Site aggregation in which site aggregation means aggregates WEB document link information accumulated in the WEB document accumulation means on a site basis, based on site aggregation rules for aggregating sites accumulated in the site aggregation rule accumulation means Steps,
A link structure analysis means for analyzing a link structure between sites aggregated by the site aggregation means;
A site evaluation step in which the site evaluation means acquires the site and link information aggregated by the site aggregation means, and obtains an evaluation value corresponding to the page rank for each site;
The site discriminating unit collects sites that are strongly related to each other based on the link structure analyzed by the link structure analyzing unit, the evaluation value for each site obtained by the site evaluating unit, and the link information. A site determination step of creating a rule and updating the site aggregation rule stored in the site aggregation rule storage means;
The WEB document evaluation means uses the link information between sites based on the site aggregation rules stored in the site aggregation rule storage means for each WEB document stored in the WEB document storage means for each document. Assigning an evaluation value and storing it in the evaluated WEB document storage means;
A large-scale WEB site evaluation method characterized by comprising:   An evaluation program for a large-scale WEB site that causes a computer to function as each means according to any one of claims 1 to 3.

Images